Image processing apparatus, image processing method, and program for attaining image processing

ABSTRACT

An image processing procedure receives mosaic image data and calculates vertical and horizontal-direction color difference components for each pixel. The image processing procedure subsequently selects an R pixel or a B pixel from the mosaic image data, and compares a variation of the vertical-direction color difference component with a variation of the horizontal-direction color difference component with regard to each of at least the selected pixels to detect edge directions of the at least selected pixels. The edge directions thus obtained are collected in an edge direction map, and then the edge directions are compared with the surrounding edge directions to remove edge noise in advance. The image processing procedure refers to the detected edge directions, and interpolates a missing color component in each pixel of the mosaic image data with the settings of one color component in each pixel in the mosaic image data.

BACKGROUND

1. Technical Field

Embodiments of the present invention relate to techniques of generatingimage data with an imaging device, such as a digital camera. Morespecifically disclosed embodiments pertain to techniques of generatingcolor image data by application of a color filter array having a mosaicarrangement of fine color filters of R (red), G (green), and B (blue)corresponding to three primary colors of light.

2. Related Art

With the advancement of digital techniques, images are increasinglyprocessed as digital data (image data). Imaging devices such as digitalcameras enable immediate output of captured images in the form of imagedata. Imaging devices are typically equipped with an electronic imagesensor consisting of small elements for converting light intensitiesinto electric signals. The imaging device focuses a captured image of asubject on the image sensor by means of an optical system and detectsthe light intensities in the individual elements as electric signals togenerate image data. The light entering the optical system may bedivided into three color components, R, G, and B, corresponding to threeprimary colors of light. The respective color lights of the three colorcomponents R, G, and B are directed to the image sensor, and theelectric signals representing the light intensities of the respectivecolor components acquired by the sensor are output to generate colorimage data. It is noted that G components are often referred to as“luminance” components whereas R and B components are often referred toas “chrominance” components.

The simplest method of acquiring the respective color lights of thethree color components R, G, and B (which are obtained as divisions ofthe light entering the optical system) by the image sensor uses aspectroscopic prism to divide the incident light into the color lightsof the three color components R, G, and B and focuses the respectivecolor lights on image sensors to generate image data with regard to therespective color components R, G, and B. This method undesirablyrequires the three image sensors. Therefore, an imaging device relyingon three image sensors to capture color images is sometimes called a“three image sensor” device.

To reduce the cost of an imaging device, one common technique uses acolor filter array to allocate one of the R, G, and B color componentsto each of the light-sensitive elements constituting the image sensor inorder to attain detection of the respective color components R, G, and Bby a single image sensor. A typical configuration of this techniqueprovides small color filters allowing transmission of only the Rcomponent in front of the photo-elements assigned for detection of the Rcomponent, small color filters allowing transmission of only the Gcomponent in front of the elements assigned for detection of the Gcomponent, and small color filters allowing transmission of only the Bcomponent in front of the elements assigned for detection of the Bcomponent. Since each element assigned for detection of a predeterminedcolor component (for example, the R component) is unable to detect theother color components (for example, the G component and the Bcomponent), the resulting image data accordingly has a mosaicarrangement of pixels of the R component, pixels of the G component, andpixels of the B component. Interpolation of missing color components ineach pixel with color components of adjacent pixels enables generationof color image data with the settings of all the color components R, G,and B in all the pixels. The process of interpolating the missing colorcomponents in the image data of the mosaic arrangement to generate colorimage data with the settings of all the color components R, G, and B issometimes referred to as a “demosaicing process”. An imaging device thatuses only one image sensor covered by a color filter array isoccasionally called a “single image sensor” device.

The single image sensor device requires the interpolation of the missingcolor components. This naturally consumes the time for interpolation andmay cause the occurrence of aliasing colors due to interpolation error.There are diverse proposed techniques with a view to preventing theoccurrence of aliasing colors while minimizing an increase of the timerequired for interpolation. One proposed technique computes colordifference components (for example, differences between the G componentand the R component) in the respective pixels after computation of themissing color components, removes the pixel with the maximum colordifference component and the pixel with the minimum color differencecomponent as noise from a pixel array of a preset number of pixelsincluding a target pixel, and recalculates the respective colorcomponents in the target pixel (see Japanese Patent Laid-Open No.2005-167974). Another proposed technique applies low-pass filters to thecolor difference components computed in the respective pixels andrecalculates the respective color components in the target pixel fromthe color difference components after removal of noise (see JapanesePatent Laid-Open No. 2005-260908).

SUMMARY

With consumers' increasing demands for higher picture quality fromimaging devices, development of demosaicing techniques that prevent theoccurrence of aliasing colors has been demanded. Since the number ofpixels constituting each image captured by imaging devices is increasingto fulfill consumers' demands for higher picture quality, development ofdemosaicing techniques that enable high-speed processing has also beendemanded.

In order to accomplish at least part of the demands mentioned above, onedisclosed embodiment is directed to an image processing apparatus forreceiving mosaic image data having settings of only one color component,R, G, or B, in each pixel and subjecting the received mosaic image datato a series of image processing to generate color image data withsettings of all the three color components, R, G, and B, in each pixel;wherein the mosaic image data is expressed as a Bayer color filter; and

the image processing apparatus comprises:

vertical-direction color difference component computation moduleconfigured to obtain the G component in the pixel location of anothercolor component or to obtain another color component in the G componentpixel location using vertically neighboring available components inorder to compute a vertical-direction color difference component betweenthe G component and the another color component in each pixel includedin each of the pixel columns of the mosaic image;

horizontal-direction color difference component computation moduleconfigured to obtain the G component in the pixel location of anothercolor component or to obtain another color component in the G componentpixel location using horizontally neighboring color components in orderto compute a horizontal-direction color difference component between theG component and the another color component in each pixel included ineach of the pixel rows of the mosaic image;

edge direction determination module configured to determine an edgedirection in a pixel by selecting from among the mosaic image data apixel in which the R component or the B component is set, and comparingan amount of variation of the vertical color difference component and anamount of variation of the horizontal color change component of theselected pixel;

edge direction map generation module configured to generate an edgedirection map in which the edge direction is set in each pixel of the Rcomponent and each pixel of the B component, by storing the determinededge direction for each the selected pixel;

edge noise removal module configured to detect edge noise that is anerroneously determined edge direction by comparing the edge directionset in the edge direction map with the edge directions of surroundingpixels, and removing the edge noise; and

missing color component interpolation module configured to interpolate amissing color component, which is a color component not set in the pixelfor each pixel of the mosaic image data, on the basis of the colorcomponent set in the mosaic image data while referencing the edgedirection map from which the edge noise has been removed.

In another embodiment, an image processing method corresponds to theimage processing apparatus described above and comprises receivingmosaic image data having settings of only one color component, R, G, orB, in each pixel and subjecting the received mosaic image data to aseries of image processing to generate color image data with settings ofall the three color components R, G, and B in each pixel; wherein theimage processing method comprises:

a first step of receiving, as the mosaic image data, image dataexpressed as a Bayer color filter;

a second step of obtaining the G component and the another colorcomponent in each pixel included in each of the pixel columns of themosaic image data, and computing a vertical-direction color differencecomponent between the G component and the another color component ineach pixel included in each of the pixel columns in the verticaldirection;

a third step of obtaining the G component and the another colorcomponent in each pixel included in each of the pixel rows of the mosaicimage data, and computing a horizontal-direction color differencecomponent between the G component and the another color component ineach pixel included in each of the pixel rows in the horizontaldirection;

a fourth step of determining an edge direction in a pixel by selectingfrom among the mosaic image data a pixel in which the R component or theB component is set, and comparing an amount of variation of the verticalcolor difference component and an amount of variation of the horizontalcolor change component of the selected pixel;

a fifth step of generating an edge direction map in which the edgedirection is set in each pixel of the R component and each pixel of theB component, by storing the determined edge direction for each theselected pixel;

a sixth step of detecting edge noise that is an erroneously determinededge direction by comparing the edge direction set in the edge directionmap with the edge directions of surrounding pixels, and removing theedge noise; and

a seventh step of interpolating a missing color component, which is acolor component not set in the pixel for each pixel of the mosaic imagedata, on the basis of the color component set in the mosaic image datawhile referencing the edge direction map from which the edge noise hasbeen removed.

In the exemplary image processing apparatus and the corresponding imageprocessing method, the procedure calculates both the vertical-directioncolor difference component in each of the pixel columns constituting thereceived mosaic image data and the horizontal-direction color differencecomponent in each of the pixel rows constituting the mosaic image data.The mosaic image data is expressed as the combination of the pixelcolumns with alternate arrangement of pixels of the G component andpixels of the R component in the vertical direction, the pixel columnswith alternate arrangement of pixels of the G component and pixels ofthe B component in the vertical direction, the pixel rows with alternatearrangement of pixels of the G component and pixels of the R componentin the horizontal direction, and the pixel rows with alternatearrangement of pixels of the G component and pixels of the B componentin the horizontal direction. Either the color difference componentbetween the G component and the R component or the color differencecomponent between the G component and the B component is obtained as thevertical-direction color difference component in each pixel column.Similarly either the color difference component between the G componentand the R component or the color difference component between the Gcomponent and the B component is obtained as the horizontal-directioncolor difference component in each pixel row. The procedure then selectsthe pixels of the R component and the pixels of the B component from themosaic image data, and compares the variation of the vertical-directioncolor difference component with the variation of thehorizontal-direction color difference component with regard to each ofat least the selected pixels to detect edge directions of the at leastselected pixels. As explained above, each of the vertical-directioncolor difference component and the horizontal-direction color differencecomponent represents either the color difference component between the Gcomponent and the R component or the color difference component betweenthe G component and the B component. In the pixels of the R component,both the vertical-direction color difference component and thehorizontal-direction color difference component are the color differencecomponents between the G component and the R component. Comparisonbetween the vertical-direction color difference component and thehorizontal-direction color difference component accordingly enablesaccurate detection of the edge direction in each of these pixels of theR component. In the pixels of the B component, both thevertical-direction color difference component and thehorizontal-direction color difference component are the color differencecomponents between the G component and the B component. Comparisonbetween the vertical-direction color difference component and thehorizontal-direction color difference component accordingly enablesaccurate detection of the edge direction in each of these pixels of theB component. A pixel in which the R component or the B component is setis then selected from among the mosaic image data, and the amount ofvariation of the vertical color difference component is compared withthe amount of variation of the horizontal color difference component,and the edge direction in the pixel is thereby determined. As describedabove, since the vertical color difference component and the horizontalcolor difference component can only be the color difference componentbetween the G component and the R component or the color differencecomponent between the G component and the B component, in a pixel inwhich the R component is set, the vertical color difference component isthe color difference component between the G component and the Rcomponent, and the horizontal color difference component is also thecolor difference component between the G component and the R component.Consequently, the direction of an edge in the pixel can be determined bycomparing the vertical and horizontal color difference components. Inthe same manner, in a pixel in which the B component is set, thevertical color difference component is the color difference componentbetween the G component and the B component, and the horizontal colordifference component is also the color difference component between theG component and the B component. Consequently, the direction of the edgecan be determined by comparing the vertical and horizontal colordifference components. An edge direction map can be generated bydetermining the edge directions for the R-component pixels and theB-component pixels. The edge direction set in the edge direction map isthen compared with the surrounding edge directions, and the edge noiseis removed, whereby an accurate edge direction map is generated. Theprocedure refers to the accurately detected edge directions andadequately interpolates the missing color component in each pixel of themosaic image data with the settings of one color component in each pixelin the mosaic image data. This arrangement thus enables generation ofadequate color image data.

Edge noise may be removed as described below in the image processingapparatus of the present invention. First, a single pixel in which theedge direction is set is selected from the edge direction map, and thepixel is set as a target pixel for determining whether the edgedirection is edge noise. A determination is then made as to whether theedge directions are the same in two pixels that are in the position ofthe second pixel in the vertical direction (upward direction or downwarddirection) from the target pixel. Alternatively, a determination is madeas to whether the edge directions are the same in two pixels that are inthe position of the second pixel in the horizontal direction (leftdirection or right direction) from the target pixel. A determination ismade that the edge direction of the target pixel is edge noise when thetwo edge directions of at least one of the pixels are the same, and theedge directions differ from the edge direction of the target pixel. Theedge direction that is determined to be edge noise may also be changedto a different edge direction.

Since natural images exhibit significant spatial correlation, edgedirections usually occur as blocks of the same edge direction.Consequently, applying the method described above makes it possible toeffectively remove an edge noise included in the edge direction map.

Alternatively, edge noise may be removed in the following manner. First,a single target pixel for determining the presence of edge noise isselected from the edge direction map. Four pixels adjacent to the targetpixel in diagonal directions are then selected, and a determination ismade as to whether the edge directions set in the four pixels are allthe same as each other and different than the edge direction of thetarget pixel. When such a target pixel (a target pixel for which theedge directions of the four pixels in diagonal directions are all thesame as each other but different than the edge direction of the targetpixel) is found, insofar as the following condition is not satisfied, adetermination is made that the edge direction of the target pixel isedge noise, and the edge direction is changed. Specifically, all thepixels within a prescribed number of pixels in the vertical direction(upward direction and downward direction) are selected, or all thepixels within a prescribed number of pixels in the horizontal direction(right direction and left direction) are selected. A determination ismade that the edge direction of the target pixel is edge noise insofaras all of the edge directions set in the selected pixels are not thesame as the edge direction of the target pixel.

The target pixel and the pixels adjacent thereto in diagonal directionshave different color components in the demosaicing image data.Consequently, when viewed in diagonal directions from the target pixel,even when the target pixel has different edge directions on both sidesthereof, the edge direction of the target pixel cannot immediately bedetermined to be edge noise. However, when the edge directions of thefour pixels diagonally adjacent to the target pixel are all the same aseach other, and the edge directions are different than the edgedirection of the target pixel, it is highly probable that the edgedirection of the target pixel is edge noise. Accordingly, in such acase, the edge directions are referenced in pixels having the same colorcomponent that are present in the vertical direction or the horizontaldirection with the target pixel at the center. Specifically, with thetarget pixel at the center, all the pixels within a prescribed number ofpixels in the vertical direction are selected, or all the pixels withina prescribed number of pixels in the horizontal direction are selected,and the edge directions of the selected pixels are referenced. In eitherthe vertical direction or the horizontal direction, if the same edgedirection is continuous including the target pixel, a determination ismade that the edge direction of the target pixel is not edge noise, butwhen the same edge direction is not continuous including the targetpixel, a determination is made that the edge direction of the targetpixel is edge noise. Edge noise can thereby be appropriately detectedand removed.

Edge noise may also be removed in the following manner. First, an areais extracted in which the amount of variation between color componentshaving the same mosaic image data is smaller than a prescribed firstthreshold value and larger than a second threshold value (which issmaller than the first threshold value). The edge noise may then beremoved with respect to the edge directions set in the pixels within thearea.

It is not necessarily impossible for a correct edge direction to bemistakenly determined to be edge noise during edge noise removal. Whensuch an erroneous determination occurs, removing the edge noise of theedge direction map may have the effect of introducing new edge noise.Therefore, rather than removing the edge noise for all areas of the edgedirection map, edge noise is removed only for areas for which edge noiseremoval is actually necessary. The risk of introducing new edge noisecan thereby be reduced. Specifically, in an area in which aliasing isunlikely to occur, since no problems occur even when edge noise ispresent, there is little need to remove the edge noise. Experimentationhas shown that aliasing is unlikely to occur in areas in which there issignificant variation between color components having the same mosaicimage data, or in areas in which there is almost no variation.Consequently, adopting a configuration in which an area is extracted inwhich the amount of variation between color components having the samemosaic image data is smaller than a prescribed first threshold value andlarger than a second threshold value (which is smaller than the firstthreshold value), and in which the edge noise is removed only for theextracted area, makes it possible to reduce the risk of introducing newedge noise into the edge direction map. Since there is no need to removeedge noise for all the areas of the edge direction map, the edge noisecan be rapidly removed from the edge direction map.

The edge noise may also be removed a plurality of times with respect tothe edge direction map from which edge noise has been removed.

Not all of the edge noise can necessarily be removed by performing theedge noise removal operation only once. For some arrangements of edgenoise, it is possible for a small amount of edge noise to remain. Forsome characteristics of the edge noise, it may also be impossible forthe edge noise to be completely removed by a single method. Even in sucha case, the edge noise can be more reliably removed by removing the edgenoise a plurality of times using the same method or a different methodfor an edge direction map from which the edge noise has been removed.

In another application, a computer reads a computer-readable mediumhaving a program for actualizing the image processing method describedabove to attain the respective functions. Another embodiment of theinvention is accordingly a computer-readable medium having program forcausing a computer to actualize a method for receiving mosaic image datahaving settings of only one color component, R, G, or B, in each pixeland subjecting the received mosaic image data to a series of imageprocessing to generate color image data with settings of all the threecolor components R, G, and B in each pixel; the program causing thecomputer to actualize:

a first function of receiving, as the mosaic image data, image dataexpressed as a Bayer color filter;

a second function of obtaining the G component and the another colorcomponent in each pixel included in each of the pixel columns of themosaic image data, and computing a vertical-direction color differencecomponent between the G component and the another color component ineach pixel included in each of the pixel columns in the verticaldirection;

a third function of obtaining the G component and the another colorcomponent in each pixel included in each of the pixel rows of the mosaicimage data, and computing a horizontal-direction color differencecomponent between the G component and the another color component ineach pixel included in each of the pixel rows in the horizontaldirection;

a fourth function for determining an edge direction in a pixel byselecting from among the mosaic image data a pixel in which the Rcomponent or the B component is set, and comparing an amount ofvariation of the vertical color difference component and an amount ofvariation of the horizontal color change component of the selectedpixel;

a fifth function for generating an edge direction map in which the edgedirection is set in each pixel of the R component and each pixel of theB component, by storing the determined edge direction for each theselected pixel;

a sixth function for detecting edge noise that is an erroneouslydetermined edge direction by comparing the edge direction set in theedge direction map with the edge directions of surrounding pixels, andremoving the edge noise; and

a seventh function for interpolating a missing color component, which isa color component not set in the pixel for each pixel of the mosaicimage data, on the basis of the color component set in the mosaic imagedata while referencing the edge direction map from which the edge noisehas been removed.

The computer reads the program according to this embodiment to attainthe respective functions described above and adequately interpolate themissing color components of the mosaic image data. This arrangement thusfacilitates the high-speed generation of the color image data whileeffectively preventing the occurrence of aliasing colors.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates the configuration of a digital cameraequipped with an image processing apparatus in one embodiment of theinvention;

FIG. 2 is a conceptual view showing the structure of a color filterarray and an image sensor included in the image processing apparatus ofFIG. 1;

FIG. 3 is a flowchart showing a color image data generation process,which is executed by the image processing apparatus of the embodiment togenerate color image data from raw image data;

FIG. 4 conceptually shows a process of converting raw image data intodata of color difference components;

FIGS. 5A and 5B show calculation formulae for calculating colordifference components from raw image data;

FIGS. 6A and 6B show low pass filters applied for removal of noiseincluded in color difference components;

FIG. 7 is a flowchart showing the details of an edge direction mapcreation process executed in the color image data generation process ofFIG. 3 by the image processing apparatus of the embodiment;

FIGS. 8A and 8B show calculation of a variation Vv in a verticaldirection from color difference components CDv in the verticaldirection;

FIGS. 9A and 9B show calculation of a variation Vh in a horizontaldirection from color difference components CDh in the horizontaldirection;

FIGS. 10A-10D show possible arrangements of adjacent pixels around atarget pixel selected for detection of an edge direction;

FIGS. 11A and 11B show one method applicable to detection of edgedirections in G pixels;

FIGS. 12A and 12B show an example of another method for computing theamount of variation of a color difference component;

FIG. 13 is a flowchart showing the edge noise removal process;

FIG. 14 is a flowchart showing the processes for removing isolated noisein the noise removal process;

FIG. 15 is a diagram showing the manner in which a determination is madeas to whether the edge direction of the target pixel is isolated noisein the isolated noise removal process;

FIG. 16 is a diagram showing an example of the manner in which isolatednoise in the edge direction map is removed;

FIG. 17 is a flowchart showing the cluster noise removal process;

FIGS. 18A and 18B are diagrams showing the manner in which adetermination is made as to whether the target pixel is cluster noise;

FIG. 19 is a flowchart showing the details of a G componentinterpolation process executed in the color image data generationprocess of FIG. 3 by the image processing apparatus of the embodiment;

FIGS. 20A-20C show a processing flow of computing the missing Gcomponent in non-G pixels;

FIG. 21 is a flowchart showing the details of a non-G pixelinterpolation process executed in the color image data generationprocess of FIG. 3 by the image processing apparatus of the embodiment;

FIG. 22 is a flowchart showing the details of a G pixel interpolationprocess executed in the color image data generation process of FIG. 3 bythe image processing apparatus of the embodiment; and

FIGS. 23A-23D show estimation of an edge direction in a G interpolationtarget pixel from edge directions of adjacent pixels.

FIG. 24 is a flowchart showing the edge noise removal process accordingto Modification 1;

FIG. 25 is a flowchart showing the process for determining the presenceof an aliasing-sensitive area in Modification 1;

FIGS. 26A-26D are diagrams showing an example of the method fordetermining, based on raw image data, whether a sharp edge is present atthe position of the target pixel;

FIGS. 27A-27G are diagrams showing an example of the method fordetermining, based on raw image data, whether a position of the targetpixel is within an adequately uniform area;

FIG. 28 is a diagram showing an example of the manner in which noise ofthe aliasing-sensitive area map is removed in Modification 2;

FIG. 29 is a flowchart showing the edge noise removal process accordingto Modification 3;

FIG. 30 is a flowchart showing the block noise removal process;

FIG. 31 is a diagram showing the manner in which the aliasing-sensitivearea map is referenced to determine whether aliasing-sensitive areascontinue in the left and right directions of the target pixel; and

FIGS. 32A-32C are diagrams showing an example of a standard fordetermining whether the edge direction of the target pixel is blocknoise.

DESCRIPTION OF EXAMPLE EMBODIMENTS

The invention may be understood more fully with reference to a preferredembodiment described below in the following sequence based on theaccompanying drawings:

A. System Configuration B. Outline of Color Image Data GenerationProcess C. Edge Direction Map Creation Process D. Edge Noise RemovalProcess:

D-1. Isolated Noise Removal Process

D-2. Cluster Noise Removal Process

E. G Component Interpolation Process F. Non-G Pixel InterpolationProcess G. G Pixel Interpolation Process H. Modifications:

H-1. Modification 1

H-2. Modification 2

H-3. Modification 3

A. System Configuration

FIG. 1 schematically illustrates the configuration of a digital camera100 equipped with an image processing apparatus 30 in one embodiment ofthe invention. As illustrated, the digital camera 100 includes anoptical system 10 that has a group of multiple lenses, an imagingassembly 20 that converts an image of a subject formed by the opticalsystem 10 into electric signals, and the image processing apparatus 30that receives the electric signals from the imaging assembly 20 andmakes the received electric signals subjected to a predetermined seriesof image processing to generate color image data.

The imaging assembly 20 has an image sensor 24 with a two-dimensionalarrangement of multiple fine imaging elements for converting the lightintensities into electric signals. A color filter array 22 is providedbefore the image sensor 24 and has a mosaic arrangement of fine colorfilters of R (red), G (green), and B (blue). The arrangement of the R,G, and B color filters constituting the color filter array 22 will bedescribed later in detail. The R color filters, the G color filters, andthe B color filters are constructed to allow transmission of R colorlight, transmission of G color light, and transmission of B color light,respectively. The image sensor 24 captures image data having a mosaicarrangement of image parts responsive to the R light intensities, imageparts responsive to the G light intensities, and image parts responsiveto the B light intensities according to the mosaic arrangement of the R,G, and B color filters in the color filter array 22.

The image processing apparatus 30 mounted on the digital camera 100receives the image data of the mosaic arrangement from the imagingassembly 20 and generates color image data with settings of the Rcomponent, the G component, and the B component in the respectivepixels. In the image processing apparatus 30 of the embodiment, a CPU, aROM, a RAM, and a data input/output interface (I/F) are interconnectedvia a bus to enable mutual data transmission. The CPU performs a seriesof processing to generate the color image data according to a programstored in the ROM. Alternatively, the program can be stored on acomputer-readable medium. Computer-readable media can include, forexample, magnetic media such as hard disks, diskettes, and magnetictape; optical media such as CD-ROMs and holographic devices; and varioushardware devices such as ASICs, PDLS, flash memory devices, and ROM andRAM devices. The resulting color image data thus generated may be outputto an external device via an external output terminal 40 or may beoutput to an external recording medium 50.

The image data with the mosaic arrangement of the R, G, and B componentscaptured by the image sensor 24 is used as source data, which isreferred to by the image processing apparatus 30 to generate the colorimage data with the settings of the R, G, and B components in therespective pixels. The image data of the mosaic arrangement captured bythe image sensor 24 may thus be hereafter referred to as “raw imagedata”.

FIG. 2 is a conceptual view showing the structure of the color filterarray 22 and the image sensor 24. As mentioned above, the image sensor24 has the two-dimensional arrangement of fine imaging elements thatoutput electric signals corresponding to the light intensities. In theillustrated example of FIG. 2, the fine imaging elements are arranged ina lattice pattern. Each of small rectangles in the lattice pattern ofthe image sensor 24 conceptually represents one imaging element (orlight-sensitive photo element).

The color filter array 22 has one of the R color filter, the G colorfilter, and the B color filter set corresponding to the position of eachof the multiple imaging elements constituting the image sensor 24. InFIG. 2, the sparsely hatched rectangles, the densely hatched rectangles,and the non-hatched open rectangles respectively denote the R colorfilters, the B color filters, and the G color filters. In thearrangement of the R, G, and B color filters, the G color filters arepositioned first to be diagonal to one another and form a checkerboardpattern. Namely the G color filters occupy half the area of the colorfilter array 22. The same numbers of the R color filters and the B colorfilters are then evenly arranged in the remaining half area of the colorfilter array 22. The resulting color filter array 22 of this arrangementshown in FIG. 2 is called the Bayer color filter array.

As mentioned above, the G color filters, the R color filters, and the Bcolor filters are designed to allow transmission of only the G colorlight, transmission of only the R color light, and transmission of onlythe B color light, respectively. The image sensor 24 accordinglycaptures the image data of the mosaic arrangement by the function of theBayer color filter array 22 located before the image sensor 24 as shownin FIG. 2. The image data of the mosaic arrangement is not processablein the same manner as ordinary image data and cannot directly express animage. The image processing apparatus 30 receives the image data of themosaic arrangement (raw image data) and generates ordinary color imagedata having the settings of the R, G, and B components in the respectivepixels.

B. Outline of Color Image Data Generation Process

FIG. 3 is a flowchart showing a color image data generation process,which is executed by the image processing apparatus 30 of the embodimentto generate color image data from the raw image data. In the structureof this embodiment, the CPU included in the image processing apparatus30 executes this color image data generation by the softwareconfiguration. This is, however, not essential, but a specific hardwareelement, for example, a signal processor, may be used for the samepurpose.

On the start of the color image data generation process, the CPU firstreceives raw image data as source data from the image sensor 24 (stepS100) and converts the received raw image data into data of colordifference components (step S102) as described below in detail.

FIG. 4 conceptually shows a process of converting raw image data intodata of color difference components. The top of FIG. 4 shows aconceptual view of the raw image data read from the image sensor 24. Theraw image data obtained with the Bayer color filter array has the mosaicarrangement of the R, G, and B components. In a horizontal direction,there are only two types of pixel rows in the raw image data: pixel rowshaving alternately arranged G pixels (pixels with G component) and Rpixels (pixels with R component) and pixel rows having alternatelyarranged G pixels and B pixels (pixels with B component). In a verticaldirection, there are similarly only two types of pixel columns in theraw image data: pixel columns having alternately arranged G pixels and Rpixels and pixel columns having alternately arranged G pixels and Bpixels. The middle of FIG. 4 conceptually shows that the raw image dataconsists of only the pixel rows of alternate G pixels and R pixels andthe pixel rows of alternate G pixels and B pixels in the horizontaldirection while consisting of only the pixel columns of alternate Gpixels and R pixels and the pixel columns of alternate G pixels and Bpixels.

Based on this result of observation, the color image data generationprocess of this embodiment converts the raw image data of the R, G, andB components into data representing differences of color components ofthe pixel rows in the horizontal direction (color difference componentsin the horizontal direction) and data representing differences of colorcomponents of the pixel columns in the vertical direction (colordifference components in the vertical direction). For example, the rawimage data is converted into data on color difference components betweenG and R components with regard to the pixel rows or the pixel columns ofalternate G pixels and R pixels. Similarly the raw image data isconverted into data on color difference components between G and Bcomponents with regard to the pixel rows or the pixel columns ofalternate G pixels and B pixels. Calculation formulae for calculatingthe color difference components will be described in detail later.

The bottom of FIG. 4 conceptually shows the color difference componentsin the horizontal direction and in the vertical direction thus obtained.A left-side view on the bottom shows color difference components CDv inthe vertical direction obtained by processing the raw image data in thevertical direction. Each of the hatched pixel columns corresponds to apixel column of alternate G pixels and B pixels. Each color differencecomponent CDv on this pixel column accordingly represents a colordifference component between the G component and the B component. Eachof the remaining non-hatched pixel columns corresponds to a pixel columnof alternate G pixels and R pixels. Each color difference component CDvon this pixel column accordingly represents a color difference componentbetween the G component and the R component. The raw image data in thehorizontal direction is similarly processable. Each of hatched pixelrows gives color difference components CDh between the G pixels and theB pixels. Each of the remaining non-hatched pixel rows gives colordifference components CDh between the G pixels and the R pixels.

FIGS. 5A and 5B show calculation formulae for calculating the colordifference components from the raw image data. FIG. 5A shows calculationformulae for calculating the color difference components CDv in thevertical direction, and FIG. 5B shows calculation formulae forcalculating the color difference components CDh in the horizontaldirection. In these calculation formulae, “z” denotes values of the rawimage data obtained by the image sensor 24, and z(r,s) shows a value ata specific position defined by an r-th pixel position downward from theposition of the origin set in an image and an s-th pixel positionrightward from the position of the origin (see the top of FIG. 4).

Referring to FIG. 5A, the description regards the calculation formulaefor calculating the color difference components CDv in the verticaldirection. When a target pixel as an object of computation of the colordifference component CDv is a pixel with the G component (G pixel), theupper calculation formula of FIG. 5A is adopted. The process firstcalculates an average of values of pixels directly above and below thetarget pixel (that is, a second term surrounded by the one-dot chainline) and subtracts the calculated average from the G component of thetarget pixel (that is, a first term surrounded by the broken line) tocalculate a color difference component CDv of the target pixel in thevertical direction. Since the target pixel for computation of the colordifference component CDv is the G pixel, the tone value z(r,s) of thetarget pixel naturally represents the G component. The element z(r−1,s)in the second term shows a tone value of the raw image data in the pixelabove the target pixel, whereas the element z(r+1,s) in the second termshows a tone value of the raw image data in the pixel below the targetpixel. As explained above with reference to FIG. 4, these two pixelsabove and below the target G pixel may be R pixels or B pixels but arealways pixels of an identical color component. The second termsurrounded by the one-dot chain line in the upper calculation formula inFIG. 5A accordingly represents either the R component or the B componentcalculated from the values of the pixels (that is, either the R pixelsor the B pixels) neighboring the target pixel in the vertical direction.Application of the upper calculation formula in FIG. 5A determines thecolor difference component CDv in the vertical direction with regard tothe pixel with the G component (G pixel), irrespective of the verticallyneighboring pixels as the R pixels or the B pixels.

It should be noted that application of the identical calculation formulato the R pixels and the B pixels as the vertically neighboring pixelsgives two different types of the color difference components CDv. Whenthe pixels neighboring the target pixel in the vertical direction arethe R pixels, the resulting color difference component CDv in thevertical direction represents the color difference component between theG component and the R component. When these two pixels are the B pixels,on the other hand, the resulting color difference component CDv in thevertical direction represents the color difference component between theG component and the B component.

The color difference components CDv in the vertical direction aresimilarly computable with regard to pixels other than the G pixels (thatis, R pixels and B pixels). For example, when a target pixel as theobject of computation of the color difference component CDv is an Rpixel, pixels directly above and below the target pixel are G pixels asexplained above with reference to FIG. 4. The average of the values ofthese two pixels is believed to specify the G component in the targetpixel as the object of computation of the color difference componentCDv. Subtraction of the tone value of the target pixel (in this case,the R component) from the specified G component determines the colordifference component CDv. Each B pixel set as the target pixel issubjected to the same series of processing. With regard to the pixelsother than the G pixels, as shown in the lower calculation formula ofFIG. 5A, subtraction of a second term surrounded by the one-dot chainline from a first term surrounded by the broken line on the right sidegives the color difference component CDv in the vertical direction.

The identical calculation formula is applicable to computation of thecolor difference components CDv in the vertical direction, with regardto both the R pixel and the B pixel set as the target pixel other thanthe G pixel. It should, however, be noted that the R pixel and the Bpixel processed as the target pixel give two different types of thecolor difference components CDv. When the target pixel is the R pixel,the resulting color difference component CDv represents the colordifference component between the G component and the R component. Whenthe target pixel is the B pixel, on the other hand, the resulting colordifference component CDv represents the color difference componentbetween the G component and the B component.

The color image data generation process of the embodiment applies thecompletely identical calculation formulae for the advanced processingwith the two different types of the color difference components CDv.Application of the identical calculation formulae desirably attains thesimplicity and the high speed of the advanced processing with the twodifferent types of the color difference components CDv, which arecomparable to the conventional simple processing. Unlike theconventional simple processing, however, this advanced processing of theembodiment enables adequate demosaicing with prevention of theoccurrence of aliasing colors.

The color difference components CDh in the horizontal direction arecomputable in the same manner as the color difference components CDv inthe vertical direction explained above. The computation of the colordifference components CDh is explained briefly with reference to FIG.5B. As shown in the upper calculation formula of FIG. 5B, subtraction ofa second term surrounded by the one-dot chain line from a first termsurrounded by the broken line on the right side gives the colordifference component CDh of the G pixel in the horizontal direction. Inthe calculation formula, the element z(r,s) represents a tone value(that is, the G component) of the raw image data in a target pixel asthe object of computation of the color difference component. The elementz(r,s−1) shows a tone value of the raw image data in a pixel on the leftof the target pixel, whereas the element z(r,s+1) shows a tone value ofthe raw image data in a pixel on the right of the target pixel. Thesetwo pixels on the left and the right of the target G pixel may be Rpixels or B pixels but are always pixels of an identical colorcomponent.

In computation of the color difference component CDh in the horizontaldirection with regard to a target pixel other than the G pixel (that is,with regard to either an R target pixel or a B target pixel), the pixelson the left and the right of the R or B target pixel are the G pixels.As shown in the lower calculation formula of FIG. 5B, subtraction of asecond term surrounded by the one-dot chain line (a tone value of thetarget pixel) from a first term surrounded by the broken line (anaverage of values of the left and the right pixels) gives the colordifference component CDh in the horizontal direction.

The identical calculation formulae are applicable to computation of thecolor difference components CDh in the horizontal direction, with regardto the target G pixel having the R pixels or the B pixels as the leftand the right pixels and with regard to both the R pixel and the B pixelset as the target pixel other than the G pixel. The computation,however, gives two different types of the color difference componentsCDh, that is, the color difference component between the G component andthe R component and the color difference component between the Gcomponent and the B component. Application of the completely identicalcalculation formulae desirably attains the simplicity and the high speedof the advanced processing with the two different types of the colordifference components CDh in the horizontal direction, which arecomparable to the conventional simple processing.

As clearly understood from the comparison between FIGS. 5A and 5B, thecalculation formulae used for computation of the color differencecomponents CDv in the vertical direction are significantly similar tothe calculation formulae used for computation of the color differencecomponents CDh in the horizontal direction. Namely the processing flowcan be standardized for computation of both the color differencecomponents CDv and CDh in the two different directions. This furtherenhances the simplicity and the high speed of the overall processing.

At step S102 in the flowchart of FIG. 3, the raw image data captured bythe image sensor 24 is converted into the color difference componentsCDv in the vertical direction and the color difference components CDh inthe horizontal direction. It is preferable, although not essential, toapply low pass filters to the color difference components CDv in thevertical direction and the color difference components CDh in thehorizontal direction for the purpose of removal of possible noise fromthese color difference components. As explained later, the color imagedata generation process of the embodiment uses the color differencecomponents for detection of the edge directions included in the image.Application of the low pass filters is generally undesirable since thelow pass filters have a tendency to blur the edges and lower thedetection accuracy of the edges. The procedure of this embodiment,however, enables detection of the edge directions with extremely highaccuracy as described later. Some blur of the edges thus does notsignificantly lower the detection accuracy, but the noise removalenables stable detection of the edge directions.

FIGS. 6A and 6B shows low pass filters applied for removal of noiseincluded in color difference components. FIG. 6A shows low pass filtersto be applied to the color difference components CDv in the verticaldirection and to be applied to the color difference components CDh inthe horizontal direction. For the better understanding, FIG. 6B showsapplication of the low pass filter to the color difference component CDhin the horizontal direction. A target pixel as an object of applicationof the low pass filter is represented by a hatched rectangle in FIG. 6B.The low pass filter of this embodiment sums up a color differencecomponent in the target pixel and color difference components in twopixels on the left and two pixels on the right of the target pixel. Thecolor difference component of the target pixel is multiplied by a weightof four pixels, so that the summed color difference componentscorrespond to eight pixels. In this case, simple 3-bit shift of the sumto the lower bit gives a color difference component of one pixel, whichis equivalent to a division of the sum by 8. Such weighting enables thehigh-speed processing with the low pass filter.

Each of the color difference components CDv in the vertical directionand the color difference components CDh in the horizontal direction mayrepresent the color difference component between the G component and theR component or the color difference component between the G componentand the B component, as explained previously. The low pass filters areapplicable to the color difference components CDv in the verticaldirection and the color difference components CDh in the horizontaldirection without taking into account such different types of the colordifference components. The color image data generation process of theembodiment thus attains the extreme simplicity and the high speed of theadvanced processing.

Referring back to the flowchart of FIG. 3, after computation of thecolor difference components CDv in the vertical direction and the colordifference components CDh in the horizontal direction in the abovemanner, the color image data generation process of the embodimentcreates an edge direction map with these data on the color differencecomponents (edge direction map creation process) (step S104). The edgedirection map shows the orientation of each edge included in the imageas either the vertical direction or the horizontal direction. Thedirection of an edge may be set in advance in all the pixels, but in thepresent embodiment, the edge direction is set for pixels (R pixels and Bpixels) other than the G pixels in order to speed up processing.

A detailed description will be given hereinafter. In the color imagedata generation process of the present embodiment, processing isperformed for interpolating color components that are missing (e.g., theR component and the B component in the case of a G pixel, the Gcomponent and the B component in the case of an R pixel, and the Gcomponent and the R component in the case of a B pixel) in the raw imagedata while the edge direction map is referenced. The inclusion of noisein the edge direction set in the edge direction map makes it impossibleto appropriately interpolate the color components missing from the rawimage data. Therefore, once the edge direction map is generated,processing (edge noise removal process) is performed for removing noise(edge noise) in the edge direction map prior to interpolation of themissing color components (step S106). The edge noise removal processwill be described in detail hereinafter.

Once the edge noise in the edge direction map is removed, the CPUdetermines the tone value of the G component in each target pixel otherthan the G pixel (that is, either the R pixel or the B pixel) byinterpolation (G component interpolation process) (step S108). The Gcomponent interpolation process refers to the edge direction map andperforms interpolation with the values of the G component in adjacentpixels to determine the adequate value of the G component in the non-Gtarget pixel. The details of this G component interpolation process willbe described later.

The CPU subsequently performs interpolation of the remaining colorcomponent in the non-G target pixel other than the G pixel (non-G pixelinterpolation process) (step S110). The G component of the non-G targetpixel, which may be the R pixel or the B pixel, has already beendetermined by interpolation at preceding step S108. The processing ofstep S108 thus interpolates the B component for the R target pixel,while interpolating the R component for the B target pixel. The pixelsother than the G pixels accordingly have obtained all the colorcomponents R, G, and B. The details of the non-G pixel interpolationprocess (interpolation of the remaining color component afterinterpolation of the G component with regard to each non-G target pixelother than the G pixel) will be described later.

On completion of the interpolation in the pixels other than the Gpixels, the CPU interpolates the missing color components (the Rcomponent and the B component) of the raw image data with regard to eachG target pixel (G pixel interpolation process) (step S112). The G pixelinterpolation process also refers to the edge direction map and performsinterpolation to determine the adequate values of the missing colorcomponents in the G target pixel. The details of this G pixelinterpolation process will be described later.

On completion of the interpolation of the missing color components inthe pixels other than the G pixels (step S108 and step S110) and theinterpolation of the missing color components in the G pixels (stepS112), all the pixels have obtained all the color components R, G, andB. The CPU subsequently outputs the obtained RGB color components ascolor image data generated from the raw image data (step S114) andterminates the color image data generation process of FIG. 3.

As described above, the color image data generation process of thisembodiment detects the edge directions based on the color differencecomponents CDv in the vertical direction and the color differencecomponents CDh in the horizontal direction to create the edge directionmap. More specifically the edge directions are detected according to thetwo different types (the color difference component between the Gcomponent and the R component and the color difference component betweenthe G component and the B component) of the color difference componentsCDv in the vertical direction and the color difference components CDh inthe horizontal direction, whereby the edge direction map is generatedwith good precision. Furthermore, executing the process for removingnoise from the edge direction map generated in this manner makes itpossible to generate an even more precise edge direction map.Application of this extremely accurate edge direction map enablesadequate interpolation of the missing color components for the raw imagedata. The edge direction map creation process, the process for removingedge noise in the edge direction map, and the interpolation of themissing color components are rather simple and are executable at thehigh speed. The above series of processing in the embodiment thusenables high-speed generation of adequate color image data from rawimage data with a mosaic arrangement of the R, G, and B colorcomponents. The following description sequentially regards the detailsof the edge direction map creation process, the process (edge noiseremoval process) for removing edge noise in the edge direction map, theG component interpolation process (interpolation of the G component inthe pixels other than the G pixels), the non-G pixel interpolationprocess (interpolation of the remaining color component in the pixelsother than the G pixels), and the G pixel interpolation process(interpolation of the missing color components in the G pixels).

C. Edge Direction Map Creation Process

FIG. 7 is a flowchart showing the details of the edge direction mapcreation process executed at step S104 in the color image datageneration process of FIG. 3 by the image processing apparatus 30 of theembodiment.

The edge direction map creation process first selects one target pixelas an object of detection of the edge direction (step S200) andcalculates a variation Vv of the color difference component in thevertical direction (step S202). The variation Vv in the verticaldirection is calculated naturally from the color difference componentsCDv in the vertical direction.

FIGS. 8A and 8B show calculation of the variation Vv in the verticaldirection from the color difference components CDv in the verticaldirection. FIG. 8A shows a calculation formula, and FIG. 8B conceptuallyshows calculation of the variation Vv in the vertical directionaccording to the calculation formula of FIG. 8A. A hatched rectangle inFIG. 8B represents the position of a target pixel. The computationprocess of the variation Vv in the vertical direction first calculatesthe square of a difference between the color difference components CDvin the target pixel and a pixel above the target pixel. Similarly thecomputation process calculates the square of a difference between thecolor difference components CDv in the target pixel and a pixel belowthe target pixel. Open arrows shown between the hatched target pixel andits two vertically neighboring pixels in FIG. 8B represent computationof the squares of the differences between the color differencecomponents CDv of the respective pixels.

After computation of the squares of the respective differences betweenthe color difference components CDv in the target pixel and the upperand the lower pixels, the computation process similarly calculates thesquares of respective differences between the color differencecomponents CDv in a pixel on the left of the target pixel and pixelsabove and below the pixel on the left of the target pixel, as well asthe squares of respective differences between the color differencecomponents CDv in a pixel on the right of the target pixel and pixelsabove and below the pixel on the right of the target pixel. FIG. 8Bshows the six squares obtained from the six pairs of pixels arrayed inthe vertical direction. The summation of these six squares is specifiedas the variation Vv in the vertical direction with regard to the targetpixel.

Referring back to the flowchart of FIG. 7, after calculation of thevariation Vv in the vertical direction, the edge direction map creationprocess calculates a variation Vh of the color difference component inthe horizontal direction (step S204). The variation Vh in the horizontaldirection is calculated in the similar manner to the calculation of thevariation Vv in the vertical direction except the use of the colordifference components CDh in the horizontal direction for thecomputation.

FIGS. 9A and 9B show calculation of the variation Vh in the horizontaldirection from the color difference components CDh in the horizontaldirection. FIG. 9A shows a calculation formula, and FIG. 9B conceptuallyshows calculation of the variation Vh in the horizontal directionaccording to the calculation formula of FIG. 9A. As shown by open arrowsin FIG. 9B, the computation process successively calculates the squaresof respective differences between the color difference components CDh ina target pixel and its two horizontally neighboring pixels (that is,pixels on the left and on the right of the target pixel), the squares ofrespective differences between the color difference components CDh in apixel above the target pixel and pixels on the left and on the right ofthe pixel above the target pixel, and the squares of respectivedifferences between the color difference components CDh in a pixel belowthe target pixel and pixels on the left and on the right of the pixelbelow the target pixel. The summation of these six squares is specifiedas the variation Vh in the horizontal direction with regard to thetarget pixel.

Referring back again to the flowchart of FIG. 7, the variation Vv in thevertical direction and the variation Vh in the horizontal directioncalculated as described above are compared each other with regard to thetarget pixel (step S206). When the variation Vh in the horizontaldirection is greater or equal than the variation Vv in the verticaldirection (step S206: yes), it is determined that an edge goes throughthe target pixel in the vertical direction. A value “d” representing theedge direction is then set equal to “1” (step S208). When the variationVv in the vertical direction is greater than the variation Vh in thehorizontal direction (step S206: no), on the contrary, it is determinedthat an edge goes through the target pixel in the horizontal direction.The value “d” representing the edge direction is then set equal to “3”(step S210). Namely the pixel with the edge direction “d” set to “1” hasthe edge in the vertical direction, whereas the pixel with the edgedirection “d” set to “3” has the edge in the horizontal direction.

The squares of the differences between the color difference componentsare calculated in determination of the variation Vv in the verticaldirection (see FIG. 8) and in determination of the variation Vh in thehorizontal direction (see FIG. 9). Such calculation enables detection ofthe edge directions without taking into account the signs of thedifferences between the color difference components. One possiblemodification may thus calculate the absolute values of respectivedifferences between the color difference components, instead ofcalculation of their squares. The summation of the absolute values withregard to the adjacent pixels gives the variation Vv in the verticaldirection or the variation Vh in the horizontal direction. The square,however, enhances the difference between the color differencecomponents, compared with the absolute value and thus increases thedetection accuracy of the edge direction. It should be understood thatthis invention is not limited to the absolute or square differences andthat any other difference or similarity measure can be used instead todetermine edge directions.

Some attention is required for the comparison between the variation Vvin the vertical direction and the variation Vh in the horizontaldirection. There are accordingly some variations in edge direction mapcreation process shown in FIG. 7 as described below.

FIGS. 10A-10D show possible arrangements of adjacent pixels around atarget pixel selected for detection of the edge direction. The adjacentpixels around the selected target pixel satisfy one of the four statesshown in FIGS. 10A through 10D. In the state of FIG. 10A, a target pixelcolumn of a target pixel and its two neighbors in the vertical directiongives the squares of respective two color difference components CD_(GR)between the G component and the R component, whereas pixel columns inthe vertical direction on the left and on the right of the target pixelcolumn give the squares of respective four color difference componentsCD_(GB) between the G component and the B component. The variation Vv inthe vertical direction is accordingly computed as the sum of the twosquares of the color difference components CD_(GR) and the four squaresof the color difference components CD_(GB). The variation Vh in thehorizontal direction is similarly computed as the sum of the two squaresof the color difference components CD_(GR) and the four squares of thecolor difference components CD_(GB). The simple comparison between thevariation Vv in the vertical direction and the variation Vh in thehorizontal direction thus enables adequate detection of the edgedirection in the target pixel.

In the state of FIG. 10B, each of the variation Vv in the verticaldirection and the variation Vh in the horizontal direction is computedas the sum of the two squares of the color difference components CD_(GB)and the four squares of the color difference components CD_(GR). Thesimple comparison between the variation Vv in the vertical direction andthe variation Vh in the horizontal direction thus enables adequatedetection of the edge direction in the target pixel.

This is, however, not satisfied in the state of FIG. 10C or in the stateof FIG. 10D. For example, in the state of FIG. 10C, the variation Vv inthe vertical direction is computed as the sum of the two squares of thecolor difference components CD_(GB) and the four squares of the colordifference components CD_(GR). The variation Vh in the horizontaldirection is, however, computed as the sum of the two squares of thecolor difference components CD_(GR) and the four squares of the colordifference components CD_(GB). Namely the variation Vv in the verticaldirection is not simply comparable with the variation Vh in thehorizontal direction. In the state of FIG. 10D, the variation Vv in thevertical direction is also not simply comparable with the variation Vhin the horizontal direction.

In order to address this problem, the amount of variation of the colordifference component may be computed using different weights for thepixel column in which the target pixel is present and the pixel columnsin which the target pixel is not present. For example, in FIG. 11A, anexample of a calculation formula is shown for calculating the variationamount Vv in the vertical direction, and FIG. 11B shows an example of acalculation formula for calculating the variation amount Vh in thehorizontal direction. In these calculation formulas, the weight in thepixel column in which the target pixel is present is set to twice theweight in the pixel columns in which the target pixel is not present.Through the use of such a calculation formula, the variation amount Vvin the vertical direction and the variation amount Vh in the horizontaldirection can be compared with each other for the pixels shown in FIG.10C and the pixels shown in FIG. 10D, and the edge direction can bedetermined. In the calculation formulas shown in FIG. 11, the weight inthe pixel column in which the target pixel is present is set to twicethe weight in the pixel columns in which the target pixel is notpresent, but the weight in the pixel columns in which the target pixelis not present may also be set to half the weight in the pixel column inwhich the target pixel is present. Alternatively, a calculation formulasuch as the one shown in FIG. 12A may also be used to calculate thevariation amount Vv in the vertical direction and the variation amountVh in the horizontal direction.

Alternatively and more simply, the edge direction may be determined onlyfor the pixel (R pixel) of FIG. 10A and the pixel (B pixel) of FIG. 10B.As previously described using FIG. 2, G pixels account for half the areaof a Bayer color filter, half the remaining area are R pixels, and theother half are B pixels. Consequently, if the edge direction isdetermined for only the R pixels and the B pixels, the determination ismade for half the number of pixels, and the edge direction map cantherefore be rapidly generated. It is, of course, impossible to obtainan edge direction by the G pixels using this method. However, since thepixels (that is, the R pixels and the B pixels) for which the edgedirection is known are arranged in a checkered flag pattern, even whenthe edge direction of the G pixels is not known, adverse effects onimage quality can be prevented through the use of an interpolationmethod.

In order to accelerate processing in the present embodiment, the methodfor determining the edge direction is employed only for the pixels shownin FIGS. 10A and 10B. Consequently, in step S200 of the edge directionmap generation process shown in FIG. 7, a pixel other than a G pixel(that is, an R pixel or a B pixel) is selected as the target pixel. Thevertical-direction variation amount Vv and the horizontal-directionvariation amount Vh of the color difference component are thencalculated for the selected target pixel (steps S202, S204), and theedge direction is determined by comparing the variation amount Vv andthe variation amount Vh (steps S206 through S210), after which adetermination is made as to whether the edge direction has beendetermined for all the target pixels (that is, the R pixels and the Bpixels) (step S212). When there is any unprocessed pixel (step S212:no), the edge direction map creation process returns to step S200 toselect another target pixel among unprocessed pixels and executes thesubsequent series of processing. On completion of detection of the edgedirections with regard to all the pixels (step S212: yes), the edgedirection map creation process of FIG. 7 is terminated.

The edge direction map creation process of the embodiment enablesdetection of the edge directions included in the raw image data to bedetermined with good precision. The reason that an accurate edgedirection map can be generated in the edge direction map generationprocess of the present embodiment will be described hereinafter.

As shown in FIGS. 2 and 4, the raw image data based on the Bayer colorfilter array have data of any color component R, G, or B at onlydiscrete pixel positions. The G component has data at the rate of 1 per2 pixels (G pixels), while the R component and the B component have dataat the rate of 1 per 4 pixels (R pixels and B pixels). Because of suchdiscrete appearance of data, it is difficult to detect the edge withhigh accuracy from a variation in tone value with regard to each of thecolor components R, G, and B.

The color difference components are accordingly used instead of the R,G, and B color components. With regard to each pixel column or pixel rowhaving an alternate arrangement of the G component and the R component,the raw image data is converted into data of color difference componentsbetween the G component and the R component. With regard to each pixelcolumn or pixel row having an alternate arrangement of the G componentand the B component, the raw image data is converted into data of colordifference components between the G component and the B component. Thebottom of FIG. 4 shows conversion of the raw image data into the colordifference components CDv in the vertical direction and the colordifference components CDh in the horizontal direction. The pixel columnsand the pixel rows having the color difference components between the Gcomponent and the B component are shown as hatched rectangles in FIG. 4.

Unlike the samples of raw data which constitute the mosaic of colorcomponents, the above conversion of raw data to color differencecomponents allows for continuous signals to be obtained in at least thedirection of the pixel columns or the direction of the pixel rows. Sincecolor difference components obtained in pixel columns (or pixel rows)that are adjacent to each other differ from each other in meaning, itmay not be meaningful to compare the values of color differencecomponents between adjacent pixel columns. Consequently, when colordifference components having the same meaning are used to detect an edge(e.g., the color difference components indicating the difference betweenthe G components and the B components indicated by hatched lines in FIG.4 are used to detect an edge), the adjacent pixel column may be skipped,and the color difference component of every other pixel column may beused.

Significant variations of either or both of the color differencecomponent between the G component and the R component and/or the colordifference component between the G component and the B component usuallyoccur in an image area having an abrupt change in lightness and/orcolor. An edge is thus detectable as changes in color differencecomponents.

The edge direction map creation process of the embodiment detects theedge directions with high accuracy based on the above consideration andaccordingly gives a highly accurate edge direction map. The concreteprocedure converts the raw image data based on the Bayer color filterarray into the color difference components CDv in the vertical directionand the color difference components CDh in the horizontal direction. Asshown in the bottom of FIG. 4, the obtained data of the color differencecomponents have an alternate arrangement of pixel columns or pixel rowsof the color difference component CD_(GR) between the G component andthe R component and pixel columns or pixel rows of the color differencecomponent CD_(GB) between the G component and the B component. Thevariations of the color difference components are then calculatedaccording to the calculation formulae of FIGS. 11A and 11B or FIGS. 8Aand 8B and 9A and 9B. The calculated variations are obtained from thecolor difference components of adjacent pixel columns or adjacent pixelrows having different types. The variations calculated even from thecolor difference components having different types are comparable forthe edge detection as described above. Such edge detection according tothe color difference components of the adjacent pixel columns or theadjacent pixel rows ensures highly accurate detection of even asmall-scale edge. The accurate detection of the edge direction in eachpixel is thus enabled by the comparison between the variation Vv in thevertical direction and the variation Vh in the horizontal direction.

As clearly understood from this explanation, in the edge direction mapcreation process of the embodiment, it is important to compute thevariation Vv in the vertical direction and the variation Vh in thehorizontal direction based on the color difference components ofadjacent pixel columns or adjacent pixel rows. In other words, it is notso important what calculation formulae are used for the computation ofthe variation Vv in the vertical direction and the variation Vh in thehorizontal direction. Diverse calculation formulae are thus applicablein place of the calculation formulae in FIGS. 11A and 11B or FIGS. 8Aand 8B and 9A and 9B. For example, the variation amount Vv of thevertical direction and the variation amount Vh of the horizontaldirection can also be calculated using a calculation formula such as theone shown in FIG. 12.

D. Edge Noise Removal Process:

As described hereinafter, in the color image data generation process ofthe present embodiment, missing components of the raw image data can beappropriately interpolated by referencing the edge direction map thusgenerated. However, satisfactory results cannot be obtained byreferencing an edge direction map that is contaminated with noise.Therefore, in the color image data generation process of the presentembodiment, when the edge direction map is generated (step S104 in FIG.3), processing (edge noise removal process) is performed for removingthe noise (edge noise) included in the edge direction map before theedge direction map is referenced to interpolate missing components ofthe raw image data.

FIG. 13 is a flowchart showing the edge noise removal process. As shownin FIG. 13, the processing for removing edge noise in the edge directionmap is composed of a process (S300) for removing isolated noise and aprocess (S320) for removing cluster noise. The term “isolated noise”used herein refers to noise that exists in isolation in only a singlepixel, or line noise having a width or a height of only a single pixel.The term “cluster noise” refers to noise that occurs in groups among aplurality of pixels, and is not isolated noise. For example, line noisehaving a width and a height of several pixels or more is classified ascluster noise.

D-1. Isolated Noise Removal Process:

FIG. 14 is a flowchart showing the processes for removing isolated noisein the noise removal process. When the isolated noise removal process(step S300) is initiated, a single pixel (target pixel) is firstselected as a target for determining the presence of noise (step S302).As previously mentioned, in the edge direction map generation process ofthe present embodiment, the edge direction is determined only for Rpixels and B pixels. Consequently, an R pixel or a B pixel is alwaysselected as the pixel (target pixel) to be the target for determiningwhether the edge direction is noise.

A determination is then made as to whether the edge direction d(r,s)obtained in the target pixel location (r,s) is equal to “1” (step S304),and when this edge direction is equal to “1” (step S304: yes), adetermination such as the following is made. First, the edge directionsof the pixels above and below the target pixel are referenced, and adetermination is made as to whether the target pixel is a pixel that issurrounded above and below by pixels in which edge direction is equal to“3” (step S306). The expression d(r −2, s) shown in the flowchartindicates the edge direction obtained at the pixel that is two pixelsabove the target pixel, and d(r+2, s) indicates the edge directionobtained at the pixel that is two pixels below the target pixel. In thepresent embodiment, due to the fact that the edge direction isdetermined only for the R pixels and the B pixels, the edge direction isobtained for every other pixel in the vertical direction. Therefore, theedge direction of the second pixel upward or the second pixel downwardof the target pixel is referenced. The edge direction also can only be“1” or “3.” Consequently, a value of “6” as the sum of d(r−2, s) andd(r+2, s) means that both of these two edge directions are equal to “3.”Therefore, when the determination made in step S306 is “yes,” the edgedirection of the target pixel is changed to “3” (step S310).Specifically, given that the edge direction obtained upward of thetarget pixel, and the edge direction obtained downward of the targetpixel are both equal to “3” and only the edge direction of the targetpixel is equal to “1,” this edge direction of the target pixel isdetermined to be edge noise and is changed from “1” to “3.”

When a determination of “no” is made in step S306, the edge directionsof the pixels to the left and right of the target pixel are referencedat this time, and a determination is made as to whether the left andright edge directions are both equal to “3.” The expression d(r, s−2)shown in the flowchart indicates the edge direction obtained at thesecond pixel to the left of the target pixel, and d(r, s+2) indicatesthe edge direction obtained at the second pixel to the right of thetarget pixel. The reason for referencing the edge directions of thepixel two pixels to the right or left of the target pixel is that inthis embodiment the edge direction is being determined only for R pixelsand B pixels. A determination is then made as to whether the sum of d(r,s−2) and d(r, s+2) is “6” (step S308), and if the sum is “6” (step S308:yes), the edge direction of the target pixel is changed to “3” (stepS310). Specifically, given that the edge direction obtained to the leftand right of the target pixel is “3” and only the edge direction of thetarget pixel is “1,” this edge direction of the target pixel isdetermined to be edge noise and is changed from “1” to “3.”

The same processing is also performed when the edge direction of thetarget pixel is determined not to be “1” (step S304: no). Specifically,since the edge direction of the target pixel is “3” in this case, adetermination is made as to whether the edge directions d(r, s−2) andd(r, s+2) obtained above and below the target pixel are both “1” (stepS312), or whether the edge directions d(r−2, s) and d(r+2, s) obtainedto the left and right of the target pixel are both “1” (step S314). Wheneach of the upward and downward edge directions is “1” (step S212: yes),or when each of the left and right edge directions is “1” (step S314:yes), a determination is made that the edge direction of the targetpixel is noise and is changed from “3” to “1” (step S316).

When a determination is made as described above as to whether the edgedirection of the target pixel is noise, and the edge direction isdetermined to be noise, once the value of the edge direction of thetarget pixel is changed (step S310, step S316), a determination is madeas to whether this processing has been performed for all of the targetpixels (step S318). When target pixels remain that are not yet processed(step S318: no), the process returns to the first step, and a single newtarget pixel is selected (step S302), after which the continuoussequence of processing is performed for the new target pixel. Thisprocessing is repeated, and when a determination is made that theprocessing has been performed for all the target pixels (step S318:yes), the isolated noise removal process shown in FIG. 14 is completed.

FIG. 15 is a diagram showing the manner in which a determination is madeas to whether the edge direction of the target pixel is isolated noisein the isolated noise removal process. In FIG. 15, the target pixel isindicated by diagonal lines. In the isolated noise removal process, theedge direction of the target pixel is first acquired, and then the edgedirections obtained above and below (the edge directions of the twopixels indicated by “∘” symbols in the drawing), or the edge directionsobtained to the left and right (the edge directions of the two pixelsindicated by “×” symbols in the drawing) are referenced. A determinationis then made as to whether the edge direction of the target pixel issurrounded from above and below or from the left and right by edgedirections having different values, and when the edge direction of thetarget pixel is surrounded by edge directions having different values,the edge direction of the target pixel is determined to be isolatednoise.

FIG. 16 is a diagram showing an example of the manner in which isolatednoise in the edge direction map is removed. FIG. 16 shows an example ofa case in which pixels for which the edge direction is “3” are presentin several places in an area in which the edge direction is “1.” Asindicated by “a” in the diagram, locations at which only a single pixelhas an edge direction of “3” are determined to be isolated noise, andthe edge direction is changed to “1.” As in the locations labeled “b”through “f” in the diagram, locations in which pixels having an edgedirection of “3” constitute a line having a width of one pixel aredetermined to be isolated noise, and the edge direction is changed to“1.” After these instances of isolated noise are removed, the edgedirection of “3” remains only in locations in which the same edgedirection continues for at least two or more pixels in both the verticaland horizontal directions, as in the locations labeled “g” and “h” inthe diagram. As a result, an edge direction map that is generally freeof isolated noise can be obtained.

The edge direction map obtained through the isolated noise removalprocess is said to be “generally” free of isolated noise for thefollowing reasons. Line-like isolated noise composed of three pixels ispresent in the location labeled “f” in FIG. 16. The reason for this isthat in this isolated noise, when the pixel surrounded by the dashedline is first selected as the target pixel, the selected pixel is theonly pixel that cannot be detected as isolated noise, and the edgedirection remains as “3.” But even in such a case, performing theisolated noise removal process once more makes it possible to completelyremove the isolated noise.

As described above, when the isolated noise removal process is performedfor the edge direction map, the isolated noise is generally removed, andareas subsequently remain in which the same edge directions are grouped,as indicated by “g” in FIG. 16, for example. Such groups of outlyingedge directions will be referred to as “cluster noise.”

After the isolated noise removal process shown in FIG. 14 is completed,and the process is returned to the edge noise removal process of FIG.13, a determination is made as to whether the edge direction at thetarget pixel is part of cluster noise, and processing (cluster noiseremoval process) for removing the noise is performed when the edgedirection in the target pixel is cluster noise (step S320 in FIG. 13).

D-2. Cluster Noise Removal Process:

FIG. 17 is a flowchart showing the cluster noise removal process. Whenthe cluster noise removal process is initiated, a single target pixel isfirst selected from the edge direction map (step S322). As previouslymentioned, in the edge direction map generation process of the presentembodiment, the edge direction is determined only for R pixels and Bpixels. Consequently, an R pixel or a B pixel is always selected as thepixel (target pixel) to be the target for determining whether the edgedirection is noise.

A determination is then made as to whether the edge direction of thetarget pixel is “1” (step S324), and when the edge direction of thetarget pixel is “1” (step S324: yes), a determination such as thefollowing is made. First, the edge directions of two pixels to the leftof the target pixel and two pixels to the right of the target pixel areadded together. In particular, the edge directions d(r, s−4), d(r, s−2),d(r, s+2), and d(r, s+4) are added together, and a determination is madeas to whether this first added value is equal to “4” (step S326) whichdenotes that these four edge directions and the edge direction of thetarget pixel have the same value (“1” in this case). When this addedvalue is not equal to “4” (step S326: no); that is, when there is evenone edge direction among d(r, s−4), d(r, s−2), d(r, s+2), and d(r, s+4)that is not equal to “1”, the edge directions of the pixels at eachcorner of the target pixel are added together. In particular, the edgedirections d(r+1, s+1), d(r−1, s−1), d(r+1, s−1), and d(r−1, s+1) areadded together, and a determination is made as to whether this secondadded value is equal to “12” (step S328) which denotes that these fouredge directions have the values different from the edge direction in thetarget pixel. When the second added value is equal to “12” (step S328:yes); that is, when each of the edge directions d(r+1, s+1), d(r−1,s−1), d(r+1, s−1), and d(r−1, s+1) is equal to “3”, the edge directionof the target pixel is determined to be cluster noise, and d(r, s) ischanged to “3” (step S330). In contrast, when the first added value isequal to “4” (step S326: yes) or when the second added value is notequal to “12” (step S328: no), the edge direction of the target pixel isdetermined to actually be “1” (that is, not cluster noise).

Returning to step S324, when the edge direction of the target pixel isnot equal to “1” (step S324: no), a determination such as the followingis made. First, the edge directions of two pixels above the target pixeland two pixels below the target pixel are added together. In particular,the edge directions d(r−4, s), d(r−2, s), d(r+2, s), and d(r+4, s) areadded together, and a determination is made as to whether this thirdadded value is equal to “12” (step S332) which denotes that these fouredge directions and the edge direction of the target pixel have the samevalue (“3” in this case). When the third added value is not equal to“12” (step S332: no); that is, when there is even one edge directionamong d(r−4, s), d(r−2, s), d(r+2, s), and d(r+4, s) that is not equalto “3”, the edge directions of the pixels at each corner of the targetpixel are added together. In particular, the edge directions d(r+1,s+1), d(r−1, s−1), d(r+1, s−1), and d(r−1, s+1) are added together, anda determination is made as to whether this fourth added value is equalto “4” (step S334) which denotes that these four edge directions havethe values different from the edge direction in the target pixel. Whenthe fourth added value is equal to “4” (step S334: yes); that is, wheneach of the edge directions d(r+1, s+1), d(r−1, s−1), d(r+1, s−1), andd(r−1, s+1) is equal to “1”, the edge direction of the target pixel isdetermined to be cluster noise, and the edge direction is changed to “1”(step S336). In contrast, when the third added value is equal to “12”(step S332: yes) or when the fourth added value is not equal to “4”(step S334: no), the edge direction of the target pixel is determined toactually be “3” (that is, not cluster noise).

FIGS. 18A and 18B are diagrams showing a conceptual view of the mannerin which the determination is made as to whether the target pixel iscluster noise. The pixel indicated by diagonal lines in the diagramrepresents the target pixel. FIG. 18A shows a case in which the edgedirection of the target pixel is “1,” and FIG. 18B shows a case in whichthe edge direction of the target pixel is “3.” As shown in the diagram,even though the edge directions of the four corner pixels diagonallyadjacent to the target pixel are all the same, the edge direction of thetarget pixel in the center is different. In such a case, it is possiblefor the edge direction of the target pixel to be cluster noise.Therefore, a determination is made as to whether the edge direction ofthe target pixel is cluster noise. The case shown in FIG. 18A will bedescribed hereinafter. Since the edge direction of the target pixel is“1,” even if this edge direction is not cluster noise, edges areexpected to be present in the left and right directions thereof, and theedge directions d are expected to be “1” even in the pixels to the leftand right. Therefore, the edge directions are referenced for the fourpixels indicated by the ∘ symbols in FIG. 18A. As shown in FIG. 18A, theedge direction of the target pixel is determined to be the correct edgedirection only when the edge directions of the abovementioned fourpixels are all “1.” In all other cases, the edge direction of the targetpixel is determined to be cluster noise, and the edge direction of thetarget pixel is changed to “3.”

With reference again to FIG. 17, once the processing described above iscompleted for the target pixel selected at the start of the processing,a determination is made as to whether the processing described above iscompleted for all the target pixels (step S338). As a result, when yetunprocessed target pixels are determined to remain (step S338: no), theprocess returns to the beginning, and a new target pixel is selected(step S322), after which the sequence of processes described above isperformed. This operation is repeated, and when processing of all thetarget pixels is eventually determined to have been completed (stepS338: yes), the cluster noise removal process shown in FIG. 17 iscompleted. The process then returns to the color image data generationprocess shown in FIG. 3 via the edge noise removal process shown in FIG.13.

As previously described using FIG. 7, the amount of variation in colordifference components is emphasized in the color image data generationprocess of the present embodiment to enable even fine edges to beprecisely detected by comparing the variation amount Vv in the verticaldirection and the variation amount Vh in the horizontal direction. Onthe other hand, even when there is only a slight difference between thevertical-direction variation amount Vv and the horizontal-directionvariation amount Vh, the edge direction is determined based on thisslight difference. In areas in which no definite edges exist, the sizerelationship between the vertical-direction variation amount Vv and thehorizontal-direction variation amount Vh may be reversed by a slightdifference in color difference components between pixels. Therefore,isolated noise or cluster noise may easily occur in such areas. However,since definite edges are not present in the first place in such areas,image quality is not adversely affected to a significant degree evenwhen an edge direction is erroneously determined. Specifically, evenwhen some edge noise is included in the edge direction map, there are nosignificant adverse effects on image quality. However, it is preferredthat a noise-free edge direction map be used as reference in order toappropriately interpolate missing color components in the raw imagedata. Therefore, in the color image data generation process of thepresent embodiment shown in FIG. 3, the edge noise removal process (stepS106) described above is performed to remove edge noise from the edgedirection map when the edge noise map is generated (step S104). Once anedge direction map that is free of edge noise is obtained, processing isperformed as described hereinafter to interpolate color componentsmissing from the raw image data while referencing the edge directionmap.

E. G Component Interpolation Process

As shown in FIG. 3, in the color image data generation process of thepresent embodiment, when the edge noise removal process is completed(step S106), processing for interpolating the G component using non-Gpixels (R pixels and B pixels) is initiated (step S108).

FIG. 19 is a flowchart showing the details of the G componentinterpolation process to interpolate the G component in the R pixels andthe B pixels. The G component interpolation process first refers to thecreated edge direction map to specify the edge direction in eachinterpolation target pixel as an object of interpolation (step S340).Specifically, in the pixel location (r, s), the edge direction refers tothe value of d(r, s). As previously mentioned, the edge direction isdetermined only for R pixels and B pixels in the present embodiment, andan edge direction is not obtained for all of the pixels. However, sincethe pixels being interpolated herein are non-G pixels (R pixels and Bpixels), an edge direction can always be acquired by referencing theedge direction map.

The G component interpolation process subsequently determines whetherthe specified edge direction is equal to “1” (step S342). When thespecified edge direction is equal to “1” (step S342: yes), a colordifference component of the interpolation target pixel is interpolatedwith the color difference components of pixels neighboring above andbelow the interpolation target pixel (step S344). When the specifiededge direction is equal to “3” (step S342: no), on the other hand, thecolor difference component of the interpolation target pixel isinterpolated with the color difference components of pixels neighboringon the left and on the right of the interpolation target pixel (stepS346). The G component interpolation process then obtains the raw imagedata of the interpolation target pixel (step S348) and adds the rawimage data to the interpolated color difference component of theinterpolation target pixel to compute the G component of theinterpolation target pixel (step S349). This series of processing isdescribed more in detail.

FIGS. 20A-20C show a processing flow of computing the missing Gcomponent in non-G pixels. The interpolation target pixel is a non-Gpixel, that is, either an R pixel or a B pixel. When the interpolationtarget pixel is the R pixel, the adjacent pixels are arranged as shownin FIG. 20A. When the interpolation target pixel is the B pixel, on theother hand, the adjacent pixels are arranged as shown in FIG. 20B. It ishere assumed that the interpolation target pixel as the object ofinterpolation of the G component is the R pixel and that the specifiededge direction is equal to “1”.

At the edge direction equal to “1”, the variation Vh of the colordifference component in the horizontal direction is greater than thevariation Vv of the color difference component in the verticaldirection. In general, interpolation in the state of a less variation(closer to the steady state) is expected to have the betterinterpolation result than interpolation in the state of a greatervariation. The color difference component in the vertical directionhaving a less variation is thus used for the computation of the Gcomponent. As clearly understood from FIG. 20A, in the R interpolationtarget pixel, both the color difference component in the verticaldirection and the color difference component in the horizontal directionare obtained by subtraction of the R component from the G component.Namely the G component of the interpolation target pixel is computablefrom either the color difference component in the vertical direction orthe color difference component in the horizontal direction. For theenhanced interpolation accuracy, the G component of the interpolationtarget pixel is computed from the color difference component in thevertical direction having a less variation, as already indicated by thevalue “1” of the corresponding edge-direction indicator in theinterpolation target pixel.

The color difference component in the vertical direction is subtractionof the R component from the G component, and the raw image data obtainedin the interpolation target pixel is the R component. Addition of theraw image data to the color difference component thus immediatelydetermines the G component in the interpolation target pixel. The colordifference component used for computation of the G component may be thecolor difference component in the interpolation target pixel calculatedaccording to the calculation formula of FIG. 5. Preferably the colordifference component used for computation of the G component may be thecolor difference component after removal of noise with the low passfilter shown in FIG. 6.

The color difference component of the interpolation target pixel may becalculated from the color difference components of adjacent pixels,instead of being directly read. This modified procedure may read thecolor difference components of the pixels neighboring above and belowthe interpolation target pixel in the vertical direction and calculatethe color difference component in the interpolation target pixel fromthe read-out color difference components. The G component interpolationprocess of FIG. 19 calculates the color difference component of theinterpolation target pixel from the color difference components of theadjacent pixels at step S344 or at step S346. Another modified proceduremay read the color difference components of the pixels in a largerneighborhood of the interpolation target pixel in order to calculate thecolor difference component in the interpolation target pixel from theread-out color difference components.

After computation of the G component in all the interpolation targetpixels as described above, the CPU terminates the G componentinterpolation process of FIG. 19 and goes back to the color image datageneration process of FIG. 3. Once the G component is interpolated, thecolor image data generation process subsequently starts the non-G pixelinterpolation process to interpolate the remaining color component inthe pixels other than the G pixels.

F. Non-G Pixel Interpolation Process

FIG. 21 is a flowchart showing the details of the non-G pixelinterpolation process. As explained previously, the non-G pixelinterpolation process interpolates the remaining color component in eachnon-G pixel, that is, the B component in the R pixel or the R componentin the B pixel, after interpolation of the G component in the non-Gpixel. The basic concept of the non-G pixel interpolation process isbriefly described, prior to the processing details.

The non-G pixel interpolation process interpolates the B component inthe R pixel, while interpolating the R component in the B pixel. Whenthe interpolation target pixel is the R pixel, four diagonally arrangedpixels adjacent to the interpolation target pixel are all B pixels asshown in FIG. 20A. The B component is obtained as the raw image data ofthese four adjacent pixels. When the interpolation target pixel is the Bpixel, on the other hand, four diagonally arranged pixels adjacent tothe interpolation target pixel are all R pixels as shown in FIG. 20B.The R component is obtained as the raw image data of these four adjacentpixels. The simplest procedure performs interpolation with the raw imagedata of the four diagonally arranged pixels adjacent to theinterpolation target pixel to determine the tone value of the remainingcolor component in the interpolation target pixel. For example, the Rcomponent in the interpolation target pixel can be obtained as averageof four raw R components available in diagonally surrounding pixels.Similarly, the B component in the interpolation target pixel can beobtained as average of four raw B components available in diagonallysurrounding pixels.

The direct interpolation of the remaining color component with the rawimage data of the adjacent pixels may cause the occurrence of aliasingcolors due to some deviation in difference from the other colorcomponent. In natural color images, the RGB data is significantlycorrelated both in the intra- and inter-channel senses. This means thatthe pixels in local image areas usually have similar color differencecharacteristics. It is accordingly preferable to interpolate the colordifference component in the interpolation target pixel with the colordifference components of the adjacent pixels and determine the tonevalue of the remaining color component in the interpolation target pixelbased on the interpolated color difference component. Thus, instead ofthe direct interpolation with the raw image data of the four diagonallyarranged pixels adjacent to the interpolation target pixel, anotheravailable procedure calculates the color difference component of theinterpolation target pixel from the color difference components of thefour adjacent pixels and determines the tone value of the remainingcolor component in the interpolation target pixel based on thecalculated color difference component of the interpolation target pixel.For example, the interpolation object is the B component for the Rinterpolation target pixel in the state of FIG. 20A. The preferableprocedure obtains the color difference components between the Gcomponent and the B component in the four diagonally arranged B pixelsadjacent to the R interpolation target pixel and uses these four colordifference components to calculate the corresponding color differencecomponent in the R interpolation target pixel. For example, the colordifference component in the interpolation target location can becalculated as average of available color difference components. Theprocedure then calculates the tone value of the remaining colorcomponent from the calculated color difference component and theinterpolated G component in the R interpolation target pixel.

The diagonally arranged pixels adjacent to the R interpolation targetpixel are all B pixels. Both the color difference components CDv in thevertical direction and the color difference components CDh in thehorizontal direction computed with regard to these adjacent pixelsaccording to the calculation formulae of FIGS. 5A and 5B are the colordifference components between the G component and the B component. Thecorresponding color difference component of the interpolation targetpixel may be calculated from the total eight color differencecomponents. One typical procedure may calculate the average of the colordifference component CDv in the vertical direction and the colordifference component CDh in the horizontal direction in each of the fourdiagonally arranged pixels adjacent to the interpolation target pixel asa representative color difference component in the pixel and determinethe corresponding color difference component of the interpolation targetpixel from the representative color difference components of the fouradjacent pixels. Another typical procedure may detect the edge directionin the interpolation target pixel and calculate the color differencecomponent of the interpolation target pixel from only the colordifference components (obtained according to the calculation formulae ofFIG. 5) of the adjacent pixels satisfying the detected edge direction.This method of determining the color difference component of theinterpolation target pixel by taking into account both the colordifference components CDv in the vertical direction and the colordifference components CDh in the horizontal direction with regard to theadjacent pixels requires the rather complicated series of processing.The calculation based on the G component and the B component in the fourdiagonally arranged pixels adjacent to the interpolation target pixel,on the other hand, enables the higher-speed determination of the colordifference component CD_(GB) in the interpolation target pixel.

Based on this basic concept, the non-G pixel interpolation process ofthe embodiment shown in the flowchart of FIG. 21 interpolates theremaining color component in each non-G pixel, after interpolation ofits G component. The concrete procedure of the non-G pixel interpolationprocess is explained with reference to the flowchart of FIG. 21.

The non-G pixel interpolation process first obtains the G components andthe raw image data of the four diagonally arranged pixels adjacent tothe interpolation target pixel (step S350). When the interpolationtarget pixel is the R pixel, the four diagonally arranged adjacentpixels are all B pixels as shown in FIG. 20A. The raw image data ofthese adjacent pixels obtained here accordingly regards the B component.When the interpolation target pixel is the B pixel, on the other hand,the four diagonally arranged adjacent pixels are all R pixels as shownin FIG. 20B. The raw image data of these adjacent pixels obtained hereaccordingly regards the R component.

The non-G pixel interpolation process subsequently calculates the colordifference components of these four adjacent pixels by subtraction ofthe G component from the raw image data and computes the average of thefour color difference components as the color difference component ofthe interpolation target pixel (step S352). In the application hereof,the color difference component generally represents subtraction of the Rcomponent or the B component from the G component as shown in FIG. 5.The calculation of step S352, however, changes the sign of the colordifference component by subtraction of the G component from the rawimage data (the R component or the B component). This saves changing thesign in the later computation.

After computation of the color difference component in the interpolationtarget pixel, the non-G pixel interpolation process obtains the Gcomponent of the interpolation target pixel (step S354). Theinterpolation target pixel is the non-G pixel, so that the raw imagedata of the interpolation target pixel does not regard the G component.The G component of the interpolation target pixel has, however, beenalready determined by the G component interpolation process of FIG. 19executed prior to this non-G pixel interpolation process.

Summation of the color difference component of the interpolation targetpixel computed at step S352 and the G component obtained at step S354determines the remaining color component in the interpolation targetpixel (step S356).

After computation of the remaining color component in all the non-Ginterpolation target pixels as described above, the CPU terminates thenon-G pixel interpolation process of FIG. 21 and goes back to the colorimage data generation process of FIG. 3. The color image data generationprocess subsequently starts the G pixel interpolation process tointerpolate the missing color components in the G pixels (step S112).

G. G Pixel Interpolation Process

FIG. 22 is a flowchart showing the details of the G pixel interpolationprocess. The G pixel interpolation process first specifies the edgedirection in each interpolation target pixel as an object ofinterpolation (step S400). The edge directions have been detected inadvance and set in the edge creation map by the edge direction mapcreation process of FIG. 7. As previously mentioned, the edge directionmay be determined for all of the pixels, but to accelerate processing inthe present embodiment, the edge direction is determined only for non-Gpixels (R pixels and B pixels), and edge directions are not determinedfor the G pixels. However, as shown in FIG. 2, since the G pixels aresurrounded by non-G pixels on the periphery thereof, and edge directionsin the non-G pixels are calculated, the edge directions in the G pixelsthat are to be interpolated can be estimated from the edge directions ofthe peripheral pixels.

FIGS. 23A-23D shows estimation of the edge direction in theinterpolation target pixel from the edge directions of the adjacentpixels. The G pixel interpolation process interpolates the missing (thatis, R and B) color components in the G pixels. The original pixels ofraw data surrounding the interpolation target pixel satisfy either of anarrangement of FIG. 23A or an arrangement of FIG. 23B. In FIGS. 23A and23B, a hatched rectangle represents the interpolation target pixel. Thisestimation is performed on the assumption that the edge directions havebeen detected for only the non-G pixels. The edge directions haveaccordingly been set in four adjacent pixels above, below, on the left,and on the right of the interpolation target pixel.

FIGS. 23C and 23D show arrangements of the edge directions “d” in thefour adjacent pixels of the interpolation target pixel. In thearrangement of the edge directions “d” of the four adjacent pixels shownin FIG. 23C, the number of the adjacent pixels having the edge direction“d” set to “1” is greater than the number of the adjacent pixels havingthe edge direction “d” set to “3”. It is accordingly adequate toestimate the edge direction “d” of the interpolation target pixel as“1”. In the arrangement of the edge directions “d” of the four adjacentpixels shown in FIG. 23D, on the other hand, the number of the adjacentpixels having the edge direction “d” set to “3” is greater than thenumber of the adjacent pixels having the edge direction “d” set to “1”.It is accordingly adequate to estimate the edge direction “d” of theinterpolation target pixel as “3”.

In this manner, the estimation procedure counts the number of theadjacent pixels having the edge direction “d” set to “1” and the numberof the adjacent pixels having the edge direction “d” set to “3” andspecifies the edge direction “d” of the greater count as the edgedirection “d” of the interpolation target pixel. Instead of counting therespective numbers of the adjacent pixels, another estimation proceduremay calculate the sum of the edge directions “d” set in the fouradjacent pixels above, below, on the left, and on the right of theinterpolation target pixel. When there are the same numbers of theadjacent pixels having the edge direction “d” set to “1” and theadjacent pixels having the edge direction “d” set to “3”, the sum of theedge directions “d” is equal to “8”. When the sum of the edge directions“d” is less than “8”, it is expected that the greater number of theadjacent pixels have the edge direction “d” set to “1”. The edgedirection “d” of the interpolation target pixel is thus specified as“1”. When the sum of the edge directions “d” is greater than “8”, on theother hand, it is expected that the greater number of the adjacentpixels have the edge direction “d” set to “3”. The edge direction “d” ofthe interpolation target pixel is thus specified as “3”. When the sum ofthe edge directions “d” is equal to “8” (when there are the same numbersof the adjacent pixels having the edge direction “d” set to “1” and theadjacent pixels having the edge direction “d” set to “3”), the edgedirection “d” of the interpolation target pixel may be specified aseither “1” or “3”.

As described above, when the edge directions “d” have been set for allthe pixels, the G pixel interpolation process of FIG. 22 simply readsthe edge direction “d” of the interpolation target pixel from the edgedirection map at step S500. When the edge directions “d” have been setfor only the non-G pixels, on the other hand, the G pixel interpolationprocess reads the edge directions of adjacent pixels adjoining to theinterpolation target pixel from the edge direction map and estimates theedge direction “d” of the interpolation target pixel at step S400.

After specification of the edge direction “d” in the interpolationtarget pixel, the G pixel interpolation process determines whether thespecified edge direction “d” is equal to “1” (step S402). When thespecified edge direction “d” is equal to “1” (step S402: yes), it isdetermined that the edge goes through the interpolation target pixel inthe vertical direction. The G pixel interpolation process accordinglyobtains the R component, the G component, and the B component of theupper and the lower adjacent pixels above and below the interpolationtarget pixel (step S404). As shown in the arrangements of FIGS. 23A and23B, the upper and the lower adjacent pixels above and below theinterpolation target pixel are always the non-G pixels. The missingcolor components of the non-G pixels have already been computed by the Gcomponent interpolation process of FIG. 19 and the non-G pixelinterpolation process of FIG. 21. The R, G, and B components of theseupper and the lower adjacent pixels are thus immediately obtainable.

When the specified edge direction “d” is not equal to “1” (step S402:no), on the other hand, it is determined that the edge goes through theinterpolation target pixel in the horizontal direction. The G pixelinterpolation process accordingly obtains the R component, the Gcomponent, and the B component of the left and the right adjacent pixelson the left and on the right of the interpolation target pixel (stepS406). Since the left and the right adjacent pixels on the left and onthe right of the interpolation target pixel are always the non-G pixels,the R, G, and B components of these left and right adjacent pixels areimmediately obtainable.

The G pixel interpolation process then calculates the color differencecomponents in the adjacent pixels as the objects of acquisition of theR, G, and B components (step S408). When the edge direction “d” of theinterpolation target pixel is equal to “1” (step S402: yes), the G pixelinterpolation process calculates the color difference components in theupper and the lower adjacent pixels. When the edge direction “d” of theinterpolation target pixel is not equal to “1” (step S406: no), on theother hand, the G pixel interpolation process calculates the colordifference components in the left and the right adjacent pixels. As inthe non-G pixel interpolation process of FIG. 21, the sign-changed colordifference components are obtained at step S408 by subtraction of the Gcomponent from the R component and by subtraction of the G componentfrom the B component. This saves changing the sign in the latercomputation of the R component and the B component in the interpolationtarget pixel.

After calculation of the color difference components in the upper andthe lower adjacent pixels or in the left and the right adjacent pixels(step S408), the G pixel interpolation process performs interpolationwith the calculated color difference components of the adjacent pixelsto compute the color difference components of the interpolation targetpixel (step S410). As mentioned above, the color difference componentscalculated in the upper and the lower adjacent pixels or in the left andthe right adjacent pixels are the sign-changed color differencecomponents (that is, the color difference components by subtraction ofthe G component from the R component and the color difference componentsby subtraction of the G component from the B component). Theinterpolated color difference components are thus the sign-changed colordifference components.

The G pixel interpolation process then obtains the G component of theinterpolation target pixel (step S412). Since the interpolation targetpixel is the G pixel, the raw image data directly regards the Gcomponent. The G pixel interpolation process subsequently computes the Rcomponent and the B component in the interpolation target pixel from theinterpolated color difference components and the obtained G component ofthe interpolation target pixel (step S414). Since the interpolated colordifference components are the sign-changed (that is, the green componentsubtracted from the red component, and the green component subtractedfrom the blue component) color difference components, simple addition ofthe G component to the interpolated color difference componentsdetermines the R component and the B component of the interpolationtarget pixel.

After computation of the missing color components (the R component andthe B component) in all the G pixels as described above, the CPUterminates the G pixel interpolation process of FIG. 22 and goes back tothe color image data generation process of FIG. 3.

On completion of the G pixel interpolation process (step S112), thedemosaicing process, that is, interpolation of the missing colorcomponents has been concluded for all the pixels. The CPU then outputsthe R, G, and B components of the respective pixels as the color imagedata generated from the raw image data (step S114) and terminates thecolor image data generation process of FIG. 3.

The digital camera 100 of the embodiment performs demosaicing of the rawimage data to generate the color image data according to the series ofprocessing described below in detail. The color image data generationprocess creates the edge direction map, prior to the demosaicingprocess. By taking into account the variations of the color differencecomponents CDv in the vertical direction and the color differencecomponents CDh in the horizontal direction, the edge directions of therespective pixels are detectable from the color difference components ofthe two different types (that is, the color difference componentsbetween the G component and the R component and the color differencecomponents between the G component and the B component). This ensureshighly accurate detection of even small-scale edges and accordinglyenables creation of the accurate edge direction map. The subsequentdemosaicing process refers to this accurate edge direction map toadequately interpolate the missing color components in the respectivepixels.

Both the edge direction map creation process and the interpolation ofthe missing color components are rather simple and are executable at thehigh speed. The series of processing in the embodiment thus enableshigh-speed generation of adequate color image data from raw image datawith a mosaic arrangement of the R, G, and B color components.

Since the edge direction map referenced by the demosaicing processing issubjected to an edge noise removal process to produce an edge-noise-freeedge direction map, the demosaicing can be performed more adequately.

H. Modifications

Various modifications may be made to the color image data generationprocess of the present embodiment described above. These modificationswill be briefly described hereinafter.

H-1. Modification 1:

In the edge noise removal process of the present embodiment describedpreviously using FIG. 13, the isolated noise removal process shown inFIG. 14 and the cluster noise removal process shown in FIG. 17 weredescribed as being performed for all of the edge directions d stored inthe edge direction map. However, it is not necessarily impossible foreven a correct edge direction to be erroneously determined to be edgenoise by the isolated noise removal process or the cluster noise removalprocess. In the event of an erroneous determination, a correct edgedirection is changed to an incorrect edge direction. The object ofremoving edge noise is to appropriately interpolate missing colorcomponents in the raw image data so that aliasing does not occur even inareas in which aliasing easily occurs. For this reason, a configurationmay be adopted in which edge noise is removed only for areas in whichaliasing easily occurs.

FIG. 24 is a flowchart showing the edge noise removal process ofModification 1. The edge noise removal process of Modification 1 isexecuted instead of the edge noise removal process (step S106) in thecolor image data generation process shown in FIG. 3. As shown in FIG.24, in the edge noise removal process of Modification 1, before thepreviously described isolated noise removal process (step S300) andcluster noise removal process (step S320) are performed, a determinationis made as to whether an area is an aliasing-sensitive area (step S600).Substantially the same processing (step S164, step S166) as theaforementioned isolated noise removal process and cluster noise removalprocess is performed only for the aliasing sensitive area (step S162:yes). For non-aliasing sensitive areas, isolated noise removal andcluster noise removal processes are not performed (step S162: no).

FIG. 25 is a flowchart showing the process for determining the presenceof an aliasing-sensitive area in Modification 1. As shown in FIG. 25,when the aliasing-sensitive area determination process is initiated, asingle target pixel is first selected (step S602). The target pixel inthis instance is a pixel used as a target for determining the presenceof an aliasing-sensitive area, and is selected from among pixels inwhich an edge direction is set in the edge direction map.

A determination is then made as to whether a sharp edge is present atthe position of the selected target pixel (step S604). Various methodscan be used to determine the presence of a sharp edge, and thedetermination in this case is made directly from the raw image data.

FIGS. 26A-26D are diagrams showing an example of the method fordetermining, based on the raw image data, whether a sharp edge ispresent at the position of the target pixel. FIG. 26A shows thedetermination formulas used in the determination, and a sharp edge isdetermined to be present when the raw image data of the target pixelsatisfy any of the determination formulas shown in FIG. 26A. Thedetermination formula shown at the top of FIG. 26A will first bedescribed. The expression z(r, s) shown in FIG. 26A indicates the rawimage data obtained at the target pixel. The expression z(r, s−2)indicates the raw image data obtained at the pixel two pixels to theleft of the target pixel, and z(r, s+2) indicates the raw image dataobtained at the pixel two pixels to the right of the target pixel. FIG.26B shows the positional relationship between these three pixels; thatis, the target pixel and the pixels to the left and right of the targetpixel.

Since the raw image data in this instance are image data obtainedthrough the use of a Bayer color filter such as the one shown in FIG. 2,the raw image data z(r, s) of the target pixel indicated by diagonallines in FIG. 26B, the raw image data z(r, s−2) of the left pixelindicated by the ∘ symbol in FIG. 26B, and the raw image data z(r, s+2)of the right pixel are all image data of the same color component.Consequently, when there is no significant change in the raw image datain at least the left-right direction in the position of the targetpixel, no significant difference is expected to exist between the rawimage data of the target pixel and the average value of the raw imagedata to the left and right of the target pixel. This principle will bedescribed with reference to FIG. 26D.

FIG. 26D shows an example of the size relationship between the raw imagedata of the target pixel, the raw image data of the pixels to the leftand right of the target pixel, and the average value of the raw imagedata of the left and right pixels. In the example shown, there is asignificant change in the raw image data between the target pixel andthe pixel on the right, but when such a large change is present, thereis a large deviation between the raw image data of the target pixel andthe average value of the raw image data of the pixels to the left andright of the target pixel. Conversely, this deviation is small when theraw image data are substantially constant, or when the raw image dataincrease or decrease monotonically. Therefore, a threshold value isdetermined in advance, and when the deviation between the raw image dataof the target pixel and the average value of the raw image data of thepixels to the left and right of the target pixel is larger than thethreshold value, the raw image data can be considered to have asignificant variation in the horizontal direction.

The determination formula shown at the top of FIG. 26A is fordetermining whether the raw image data varies significantly in thehorizontal direction as described above. The square of the deviation anda prescribed threshold value T are compared in the determination formulaso that the determination can be made without distinguishing betweenpositive and negative variation amounts. When the square of thedeviation is larger than the threshold value T, the raw image data isconsidered to significantly change in the horizontal direction, and asharp edge is therefore determined to be present at the target pixel.

The determination formula shown at the bottom of FIG. 26A is fordetermining whether the raw image data varies significantly in thevertical direction. Specifically, z(r−2, s) in the lower determinationformula indicates the raw image data obtained at the pixel two pixelsupward from the target pixel, and z(r+2, s) indicates the raw image dataobtained at the pixel two pixels downward from the target pixel. FIG.26C shows the positional relationship between the target pixel and thepixels above and below the target pixel. Among these three pixels, sinceraw image data of the same color component is obtained, no significantdifference is expected between the raw image data of the target pixeland the average value of the image data of the pixels indicated by the ×symbols in FIG. 26C if there is no significant change in the raw imagedata in at least the vertical direction at the position of the targetpixel. Consequently, when the lower determination formula shown in FIG.26A holds true, the raw image data is considered to significantly changein the vertical direction, and a sharp edge is therefore determined tobe present at the target pixel.

In step S604 of the aliasing-sensitive area determination process shownin FIG. 25, a determination is made directly from the raw image datathrough the use of the determination formulas shown in FIG. 26A as towhether a sharp edge is present at the target pixel. Experimentation hasshown that aliasing is unlikely to occur in locations at which a sharpedge is present. Therefore, when a sharp edge is determined to bepresent (step S604: yes), since the target pixel is not considered to bean aliasing-sensitive area, the value “1” indicating the absence of analiasing-sensitive area is set as an aliasing indicator c for indicatingwhether an aliasing-sensitive area is present (step S610). The methodfor determining whether a sharp edge is present at the target pixel isnot limited to the method shown in FIG. 26, and various methods may beused.

When a sharp edge is determined to not be present in the target pixel(step S604: no), a determination is then made as to whether the targetpixel is positioned within an adequately uniform area (step S606). Thedetermination as to whether the target pixel is positioned within anadequately uniform area can also be performed using various methods, thesame as the determination described above regarding the presence or alarge edge, but the determination is made directly from the raw imagedata in this instance as well.

FIGS. 27A-27G are diagrams showing an example of the method fordetermining, based on the raw image data, whether the position of thetarget pixel is within an adequately uniform area. The basic principleunderlying this determination will first be described with reference toFIG. 27A. The pixel indicated by diagonal lines in the drawing is thetarget pixel. As previously mentioned, since the target pixel isselected from pixels for which the edge direction is being calculated,and this edge direction is calculated only for the R pixels and Bpixels, the target pixel is an R pixel or a B pixel. As shown in FIG. 2,the raw image data are image data obtained through the use of a Bayercolor filter. Consequently, the raw image data obtained at the fourpixels adjacent to the target pixel above and below and to the left andright are all G-component image data. Since the target pixel is in anadequately uniform area, the raw image data at the four pixels adjacentabove and below and to the left and right are expected to havesubstantially the same value. Therefore, a determination is made as towhether the values of the raw image data at the pixels indicated by the∘ symbols in FIG. 27A are similar to each other, and when the values aresimilar, the target pixel can be determined to be present in anadequately uniform area.

Various methods can be used to determine whether the four pixelsindicated by the ∘ symbols in FIG. 27A have similar raw image data. Forexample, according to the simplest method, the difference between themaximum value and the minimum value of the raw image data of the fourpixels is found, and the four sets of raw image data can be determinedto have similar values if the difference is adequately small. However,the determination in the present example is made using a method such asthe one described below.

FIG. 27B shows the determination formulas used in Modification 1 todetermine whether the position of the target pixel is within anadequately uniform area. Four determination formulas are shown in FIG.27B, but when all of these determination formulas are satisfied, thetarget pixel is determined to be in a uniform area. The topmostdetermination formula in FIG. 27B will first be described.

The expression z(r, s−1) in the determination formula at the top of FIG.27B indicates the raw image data of the pixel adjacent to the left ofthe target pixel, z(r−1, s) indicates the raw image data of the pixeladjacent above the target pixel, and z(r+1, s) indicates the raw imagedata of the pixel adjacent below the target pixel. FIG. 27C shows thepositional relationship between these three pixels and the target pixel.The raw image data of the pixel indicated by the ∘ symbol, and theaverage value of the raw image data of the upper and lower pixelsindicated by the × symbols in FIG. 27C are expected to havesubstantially the same value. Therefore, the square of the differencethereof is calculated, and when the square value is smaller than aprescribed threshold value α, the possibility is high that the raw imagedata of the pixel indicated by the ∘ symbol, and the two pixelsindicated by the × symbols in FIG. 27C have substantially the samevalue. However, it is possible that although the raw image data of thepixels indicated by × in the drawing may have substantially the samevalue, the average value thereof may happen to be substantially the sameas that of the raw image data of the ∘ pixels. Therefore, in order toeliminate such a possibility the determination is made in the samemanner in a different orientation.

The second determination formula from the top in FIG. 27B is used tomake the same determination as the top determination formula in adifferent orientation. FIG. 27D is a diagram showing the pixel positionsreferenced by the determination formula listed second from the top. Inthis determination as well, the difference is calculated between theaverage value of the raw image data at the pixels indicated by × symbolsin FIG. 27D, and the raw image data at the pixel indicated by ∘, and thesquare of the difference is compared with the prescribed threshold valueα. When the square of the difference is smaller than the threshold valueα, the raw image data of the three pixels shown in FIG. 27D areconsidered to have substantially the same value. Confirmation isperformed in the same manner for the determination formula listed thirdfrom the top, and the determination formula listed at the bottom in FIG.27B. FIG. 27E shows the pixel positions referenced by the determinationformula listed third from the top, and FIG. 27F shows the pixelpositions referenced by the determination formula listed at the bottom.In the case of FIG. 27E or FIG. 27F as well, the difference iscalculated between the average value of the raw image data at the pixelsindicated by × symbols, and the raw image data at the pixel indicated by∘, and the square of the difference is compared with the prescribedthreshold value α. As a result, when the four determination formulasshown in FIG. 27B are all satisfied, the raw image data of the fourpixels indicated by ∘ in FIG. 27A are all considered to havesubstantially the same value, and the target pixel can therefore bedetermined to be in an adequately uniform area.

The threshold value α was described as a constant value in thedescription given above, but an appropriate threshold value α may alsobe computed and used according to the raw image data of the pixelssurrounding the target pixel. FIG. 27G shows an example of a typicalformula for calculating the threshold value α.

In step S606 of the aliasing-sensitive area determination process shownin FIG. 25, a determination is made in the manner described above as towhether the target pixel is present in an adequately uniform area.Experimentation has shown that aliasing is unlikely to occur in anadequately uniform area. Consequently, when the target pixel has beendetermined to be in an adequately uniform area (step S606: yes), sincethe target pixel is considered to not be in an aliasing-sensitive area,the value “1” indicating the absence of an aliasing-sensitive area isset as the aliasing indicator c (specifically, c(r, s) in the (r,s)target pixel location) for indicating whether an aliasing-sensitive areais present (step S610).

In contrast, when the position of the target pixel is determined to notbe in an adequately uniform area (step S606: no), the value “0”indicating the presence of an aliasing-sensitive area is set as thealiasing indicator c for indicating whether an aliasing-sensitive areais present (step S608). Specifically, as previously mentioned,experimentation has shown that aliasing is unlikely to occur in areas inwhich there are sharp edges, or areas that are adequately uniform.Therefore, the occurrence of aliasing is determined to be possible forother areas, and the value “0” indicating the presence of analiasing-sensitive area is set as the aliasing indicator c forindicating whether an aliasing-sensitive area is present.

Once the presence or absence of an aliasing-sensitive area is determinedas described above for the target pixel selected in step S602, thealiasing-sensitive area determination process shown in FIG. 25 iscompleted, and the process returns to the edge noise removal process ofModification 1 shown in FIG. 24. As shown in FIG. 24, in the edge noiseremoval process of Modification 1, when the process returns from thealiasing-sensitive area determination process, a determination is madeas to whether the target pixel is within an aliasing-sensitive area(step S162). The target pixel in this instance is the pixel selected instep S602 of FIG. 25. As previously described using FIG. 25, thealiasing indicator c is set to “0” (corresponding to step S162: yes)when the target pixel is within an aliasing-sensitive area, and thealiasing indicator c is set to “1” (corresponding to step S162: no) whenthe target pixel is not within an aliasing-sensitive area. Consequently,in step S162 of FIG. 24, the determination as to whether the targetpixel is within an aliasing-sensitive area can immediately be made byreferencing the value of the aliasing indicator c.

When the target pixel is determined not to be within analiasing-sensitive area (step S162: no), the process skips ahead to adetermination as to whether the processing has been completed for allthe pixels (target pixels) for which edge directions are set in the edgedirection map (step S168 of FIG. 24). In the aliasing-sensitive areadetermination process, a new target pixel is selected (step S602 of FIG.25), a determination is made as to whether the target pixel is in analiasing-sensitive area, the aliasing indicator c is set to thedetermination result, and the process then returns again to the edgenoise removal process of FIG. 24. In the edge noise removal process, adetermination is made as to whether the target pixel is within analiasing-sensitive area (step S162 of FIG. 24). This operation isrepeated, and when a target pixel that is in an aliasing-sensitive areais found (step S162: yes), the isolated noise removal process (stepS164) and the cluster noise removal process (step S166) are performed.

The isolated noise removal process (step S164) and cluster noise removalprocess (step S166) performed in the edge noise removal process ofModification 1 are substantially the same as the isolated noise removalprocess (step S300) previously described using FIG. 14 and as thecluster noise removal process (step S320) previously described usingFIG. 17, respectively, and differ only in the following aspect.Specifically, in the isolated noise removal process (S300) shown in FIG.14 or the cluster noise removal process (S320) shown in FIG. 17, atarget pixel is selected at the beginning of each process, and at theend of each process, a determination is made as to whether processinghas been completed for all the target pixels. In contrast, the isolatednoise removal process (step S164) and the cluster noise removal process(step S166) of Modification 1 differ from the abovementioned processesonly in that the abovementioned selection and determination are notperformed.

An overview of the isolated noise removal process (step S164) ofModification 1 will be given hereinafter with reference to the flowchartof FIG. 14. A target pixel is selected first (step S302) in theflowchart of FIG. 14, but such a process is not performed in theisolated noise removal process (step S164) of Modification 1. The reasonfor this is that the isolated noise removal process (step S164) ofModification 1 is executed for the target pixel when the target pixelselected in the aliasing-sensitive area determination process isdetermined to be within an aliasing-sensitive area (step S162: yes), asshown in FIG. 24.

When the isolated noise removal process (step S164) of Modification 1 isinitiated, a determination is made as to whether the edge direction ofthe target pixel is “1” or “3”. When the edge direction is “1”(corresponding to step S304 of FIG. 14), determination is made as towhether both the upward and downward edge directions are “3”(corresponding to step S306 of FIG. 14), or both the left and right edgedirections are “3” (corresponding to step S308 of FIG. 14). When any oneof these alternatives holds true, isolated noise is determined to bepresent, and the edge direction of the target pixel is changed to “3”(corresponding to step S310 of FIG. 14).

In the same manner when the edge direction of the target pixel is “3,” adetermination is made as to whether both the upward and downward edgedirections are “1” (corresponding to step S312 of FIG. 14), or both theleft and right edge directions are “1” (corresponding to step S314 ofFIG. 14). When any one of these alternatives holds true, isolated noiseis determined to be present, and the edge direction of the target pixelis changed to “1” (corresponding to step S316 of FIG. 14).

In the isolated noise removal process (step S164) of Modification 1shown in FIG. 24, once a determination such as the one described aboveis made, the process is immediately ended, and the process returns tothe edge noise removal process shown in FIG. 24. The cluster noiseremoval process (step S166) is then initiated. An overview of thecluster noise removal process (step S166) of Modification 1 will begiven hereinafter with reference to the flowchart of FIG. 17.

A target pixel is selected at the beginning (step S322) in the flowchartof FIG. 17 as well, but such a process is not performed in the clusternoise removal process (step S166) of Modification 1. The reason for thisis that the cluster noise removal process (step S166) of Modification 1is a process subsequently executed for the target pixel selected in theisolated noise removal process (step S164) of Modification 1 describedabove.

When the cluster noise removal process (step S166) of Modification 1 isinitiated, a determination is first made as to whether it is possiblethat the edge direction of the target pixel is the aforementionedcluster noise. When cluster noise is determined to be potentiallypresent, a determination is made according to the edge direction of thetarget pixel as to whether cluster noise is present by referencing theedge directions of pixels in the vertical direction or the horizontaldirection, and when cluster noise is determined to be present, the edgedirection of the target pixel is changed. These steps are performed asdiscussed above in connection with FIG. 17.

In the cluster noise removal process (step S166) of Modification 1 shownin FIG. 24, once such operations as described above are performed, theprocess is immediately ended, and the process returns to the edge noiseremoval process shown in FIG. 24. A determination is then made as towhether the sequence of processes described above has been completed forall the pixels (target pixels) for which edge directions are set in theedge direction map (step S168 of FIG. 24). As a result, when unprocessedtarget pixels remain (step S168: no), the process again returns to thebeginning of the edge noise removal process (step S106 of FIG. 3) ofModification 1, and the aliasing-sensitive area determination processpreviously described using FIG. 25 is restarted (step S600). In thealiasing-sensitive area determination process, a new target pixel isselected, and the sequence of processes described above is performed forthe selected target pixel. This operation is repeated, and once theprocesses are finally completed for all the target pixels (step S168 ofFIG. 24: yes), the edge noise removal process of Modification 1 iscompleted.

In the edge noise removal process of Modification 1 described above, analiasing-sensitive area (area in which aliasing easily occurs) isdetected, and the isolated noise removal process and cluster noiseremoval process are performed only for the detected area. It istherefore possible to prevent the isolated noise removal process and thecluster noise removal process from being unnecessarily performed forareas in which aliasing is unlikely to occur, and correct edgedirections can be prevented from being erroneously determined to be edgenoise. As a result, the edge noise removal process of Modification 1makes it possible to obtain a more accurate edge direction map, andmissing color components in the raw image data can interpolated whilesuch an edge direction map is referenced, thereby making it possible toexecute demosaicing more appropriately.

H-2. Modification 2:

In Modification 1 described above, a determination is made as to whetherthe target pixel is in an aliasing-sensitive area, and when the targetpixel is in an aliasing-sensitive area, the isolated noise removalprocess (step S164 of FIG. 24) and the cluster noise removal process(step S166) are performed. However, there is potential for error in thedetermination as to whether the target pixel is in an aliasing-sensitivearea. Therefore, a configuration may be adopted in which the aliasingindicator c for indicating the result of determining whether the targetpixel is in an aliasing-sensitive area is stored in advance in a map,and processing is performed for removing noise (an erroneouslydetermined aliasing indicator) from the map (aliasing-sensitive areamap).

FIG. 28 is a diagram showing an example of the manner in which noise inthe aliasing-sensitive area map is removed in Modification 2. In theexample shown, the processing for removing noise in thealiasing-sensitive area map is divided into a first stage, a secondstage, and a third stage. The diagram shows the manner in which noise isdetected in each stage.

In the first stage, a location is detected in which the aliasingindicator is erroneously determined to be “1” (aliasing-sensitive areanot present), and processing is performed for changing the aliasingindicator c to “0” (aliasing-sensitive area present). The location atwhich the aliasing indicator is erroneously determined to be “1” isdetected as follows. First, a single pixel is selected in which thealiasing indicator c is set to the value of “1” in thealiasing-sensitive area map. As previously described using FIG. 25,since the determination as to whether an aliasing-sensitive area ispresent is performed for a pixel for which the edge direction iscalculated, the aliasing indicators are set in a checkered flag patternin the aliasing-sensitive area map. Therefore, a single pixel in whichthe value of “1” is set is selected from among the aliasing indicatorsset in the aliasing-sensitive area map, and the selected pixel is usedas the determination object (target pixel).

The aliasing indicators c that are set on the periphery of the selectedtarget pixel are then referenced. As described above, since the aliasingindicators c are set in a checkered flag pattern in thealiasing-sensitive area map, the aliasing indicators are set in fourcorner positions adjacent in diagonal directions on the periphery of thetarget pixel. When the values of the aliasing indicators set in the fourcorners are all “0,” a determination is made that the aliasing indicatorc of the target pixel has been erroneously set to “1.” The portionlabeled “First Stage” in FIG. 28 shows a case in which the aliasingindicator of the target pixel is determined to have been erroneously setto “1” (that is, the target pixel is noise). As shown in the drawing,the aliasing indicator of the target pixel is “1,” but the aliasingindicators of the four adjacent corner pixels are all “0.” In such acase, since the aliasing indicator of the target pixel is considered tohave been erroneously set to “1,” the value of the aliasing indicator ofthe target pixel is changed to “0.” In the first stage, the operationdescribed above is performed for all pixels in the aliasing-sensitivearea map for which the value of the aliasing indicator c is set to “1.”

A pixel for which the value of the aliasing indicator is erroneously setto “1 (aliasing-sensitive area not present) is detected in the firststage as described above, but in the second and third stages, processingis performed for detecting a pixel for which the aliasing indicator iserroneously set to “0 (aliasing-sensitive area present),” and thealiasing indicator is changed to “1 (aliasing-sensitive area present).”

In the second stage, a pixel for which the value of the aliasingindicator c is set to “0” in the aliasing-sensitive area map is selectedas the target pixel. The aliasing indicators at the pixels adjacent atthe four corners of the target pixel are read, and a determination ismade as to whether three or more aliasing indicators among these fouraliasing indicators are set to “1.” As a result, when three or morealiasing indicators are “1,” the aliasing indicator of the target pixelis determined to have been erroneously set to “0,” and the value of thealiasing indicator of the target pixel is changed to “1.” In the portionlabeled “Second Stage” in FIG. 28, an example is shown of a case inwhich the aliasing indicator of the target pixel is determined to havebeen erroneously set to “0” (that is, the target pixel is determined tobe noise). As shown in the diagram, the aliasing indicator of the targetpixel is “0,” but a larger number of values has been set to “1” in theadjacent four corner pixels. In such a case, the aliasing indicator c ofthe target pixel is determined to have been erroneously set to “0,” andthe value of the aliasing indicator is changed to “1.” In the secondstage, the operation described above is performed for all pixels forwhich the value of the aliasing indicator in the aliasing-sensitive areamap is set to “0.”

A pixel for which the value of the aliasing indicator c is set to “0” inthe aliasing-sensitive area map is selected as the target pixel in thethird stage as well. However, acquired in the third stage are fouraliasing indicators c that are set in pixels located four pixels awayfrom the target pixel in the vertical and horizontal directions. FIG. 28shows the manner in which the aliasing indicators c set in these fourpixels are acquired in the portion labeled “Third Stage.” The pixelindicated by diagonal lines in the drawing is the target pixel, and thepixels located four pixels away from the target pixel in the verticaland horizontal directions are indicated by ∘ symbols. A determination ismade as to whether three or more aliasing indicators are set to “1”among the four aliasing indicators set in the pixels indicated by ∘symbols. As a result, when three or more aliasing indicators are “1,”the aliasing indicator of the target pixel is determined to have beenerroneously set to “0,” and the value of the aliasing indicator ischanged to “1.” In the example shown, the aliasing indicator of thetarget pixel is determined to have been erroneously set to “0,” and thevalue of the aliasing indicator of the target pixel is changed to “1.”In the third stage, the operation described above is performed for allpixels for which the value of the aliasing indicator in thealiasing-sensitive area map is set to “0.”

In Modification 2 as described above, pixels that are present in analiasing-sensitive area are detected, an aliasing-sensitive area map isgenerated, and noise in the map is removed by performing the threestages of processing described above for the aliasing-sensitive areamap. As a result, an aliasing-sensitive area map can be obtained inwhich the presence or absence of aliasing-sensitive areas is correctlydetermined. Such an aliasing-sensitive area map is then referenced, andthe previously described isolated noise removal process (step S300 orstep S164) and the previously described cluster noise removal process(step S320 or step S166) are performed only for the edge directions ofpixels that are determined to be in an aliasing-sensitive area. It istherefore possible to prevent the isolated noise removal process and thecluster noise removal process from being performed for areas in whichaliasing is unlikely to occur, and correct edge directions can beprevented from being erroneously determined to be edge noise. As aresult, a more correct edge direction map can be obtained. Missing colorcomponents in the raw image data can be interpolated while such an edgedirection map is referenced, thereby making it possible to executedemosaicing more appropriately.

H-3. Modification 3:

The isolated noise removal process (step S300 or step S164) and thecluster noise removal process (step S320 or step S166) described abovereference relatively narrow areas and remove edge noise. Consequently,in a case in which edge noise appears to be present insofar as narrowareas are observed individually, but in which blocks of edge noise areapparent when a wider range is observed (an example of which is shown inFIG. 16( h)), it may be impossible to effectively remove such blocks(large blocks) of edge noise. The term “block noise” will be usedhereinafter to refer to individual units of edge noise that constitutesuch large blocks of edge noise. Such a problem can be overcome by theoperations described below.

FIG. 29 is a flowchart showing the edge noise removal process ofModification 3. The edge noise removal process of Modification 3 shownin the diagram differs from the edge noise removal process previouslydescribed using FIG. 13 in that after the isolated noise removal process(step S300) and cluster noise removal process (step S320), a block noiseremoval process (step S700) is performed, and then the isolated noiseremoval process (step S300) is performed again. Since the isolated noiseremoval process (step S300) and the cluster noise removal process (stepS320) are exactly the same as the processes previously described, nodescription thereof will be given, and the description will be of theblock noise removal process.

In FIG. 29, the isolated noise removal process (step S300) is performedonly once after the block noise removal process (step S700), but thisconfiguration is not limiting, and the isolated noise removal processmay be performed a plurality of times. Alternatively, the edge noiseremoval process may be ended without performing the isolated noiseremoval process (step S300) after the block noise removal process (stepS700). A supplementary description of this aspect will be given afterthe block noise removal process (step S700) is described.

FIG. 30 is a flowchart showing the block noise removal process. Aspreviously mentioned, this process is performed for the edge directionmap for which the isolated noise removal process (step S300) and clusternoise removal process (step S320) have been performed in the edge noiseremoval process of Modification 3.

When the block noise removal process (step S700) is initiated, a singletarget pixel in which the edge direction is set is first selected fromthe edge direction map (step S702). A determination is then made as towhether an aliasing-sensitive area is continuous to the left or theright of the selected target pixel (step S704). This determination ismade by referencing an aliasing-sensitive area map generated in advance.

FIG. 31 is a diagram showing the manner in which the aliasing-sensitivearea map is referenced to determine whether an aliasing-sensitive areais continuous in the left and right directions of the target pixel. Thepixel indicated by diagonal lines in the diagram is the target pixel,and a determination is made as to whether the pixels in the areaenclosed by the dashed line rectangle are all an aliasing-sensitivearea, that is, whether a “0” is set for all of these pixels in thealiasing-sensitive area map. In FIG. 31, the presence or absence of analiasing-sensitive area is determined for the areas made up of sixpixels in the left direction and six pixels in the right direction, butit is possible to change the size of the area. When the pixels withinthese areas all constitute an aliasing-sensitive area, analiasing-sensitive area is determined to be continuous in the left andright directions of the target pixel (step S704: yes). In contrast, whenthere is even one pixel that is not an aliasing-sensitive area, thealiasing-sensitive area is determined not to be continuous (step S704:no).

The aliasing-sensitive area map is referenced, and when analiasing-sensitive area is determined to be continuous in the left andright directions of the target pixel (step S704: yes), a determinationis then made as to whether the edge direction of the target pixel isblock noise by referencing the edge direction map (step S706). Thisdetermination is made as described below.

FIGS. 32A-32C are diagrams showing an example of a standard fordetermining whether the edge direction of the target pixel is blocknoise. The presence or absence of block noise is determined byconfirming whether edge directions that are different than the edgedirection of the target pixel are set in the pixels to the left andright of the center target pixel. In FIG. 32, the target pixel isindicated by diagonal lines, and the pixels for confirming the edgedirections are indicated by ∘ symbols. The edge directions of the pixelsto the left and right of the center target pixel are confirmed becausethe fact that an aliasing-sensitive area is continuous to the left andright of the target pixel was already confirmed in step S704 of FIG. 30.

For example, in the example shown in FIG. 32A, the edge directions ofthe fourth and sixth pixels to the left of the target pixel, and theedge directions of the fourth and sixth pixels to the right of thetarget pixel are referenced. When all of the edge directions set inthese four pixels differ from the edge direction set in the targetpixel, the edge direction of the target pixel is determined to be blocknoise. In the example shown in FIG. 32B, the edge directions of thesixth and eighth pixels to the left of the target pixel, and the edgedirections of the sixth and eighth pixels to the right of the targetpixel are referenced. When all of these edge directions differ from theedge direction of the target pixel, the edge direction of the targetpixel is determined to be block noise. In the same manner in the exampleshown in FIG. 32C, the edge directions of the eighth and tenth pixels tothe left and right of the target pixel are referenced, and when all ofthese edge directions differ from the edge direction of the targetpixel, the edge direction of the target pixel is determined to be blocknoise.

In step S706 of the block noise removal process shown in FIG. 30, wherethere is a correspondence to even one of the three determinationstandards shown in FIG. 32, the edge direction of the target pixel isdetermined to be block noise (step S706: yes). In the determination ofthe presence or absence of block noise, determination standards such asthose used to reference the edge directions farther to the left andright of the target pixel, for example, may also be used besides thedetermination standards shown in FIG. 32. Alternatively, it is alsopossible to use determination standards whereby three edge directionseach to the left and right of the target pixel are referenced. When theedge direction of the target pixel is determined to be block noise (stepS706: yes), the value of the edge direction is changed (step S708).Specifically, an edge direction of “1” for the target pixel is changedto “3,” and an edge direction of “3” for the target pixel is changed to“1.”

In contrast, when the edge direction of the target pixel is determinednot to be block noise (step S706: no), or when the determination isinherently made that an aliasing-sensitive area is not continuous to theleft and right of the target pixel (step S704: no), the same operationis then performed for the vertical direction. Specifically, adetermination is first made as to whether an aliasing-sensitive area iscontinuous upward and downward from the target pixel (step S710). Instep S704 described previously with reference to FIG. 31, adetermination is made as to whether all of the pixels in the area (thearea indicated by the dashed line rectangle in FIG. 31) that extends sixpixels to the left and right of the target pixel are in analiasing-sensitive area. However, in step S710, a determination is madeas to whether all of the pixels in the area that extends six pixelsabove and below the target pixel are in an allasing-sensitive area.

When all of the pixels are determined to be in an aliasing-sensitivearea (step S710: yes), the presence or absence of a correspondence tothe same determination standards as those shown in FIGS. 32A-32C isdetermined while the edge directions set in the pixels above and belowthe target pixel are referenced. Specifically, according to thedetermination standards corresponding to FIG. 32A, the edge directionsset in the four pixels that are the sum of the fourth and sixth pixelsin each of the upward and downward directions from the target pixel arereferenced, and when all of the edge directions differ from the edgedirection of the target pixel, the edge direction of the target pixel isdetermined to be block noise. In the same manner for FIGS. 32B and 32C,a determination is made as to whether the edge direction of the targetpixel is block noise on the basis of the corresponding determinationstandards. When there is a correspondence to any of these determinationstandards, the edge direction of the target pixel is determined to beblock noise (step S712: yes), and the value of the edge direction ischanged (step S714).

In contrast, when the edge direction of the target pixel is determinedto not be block noise (step S712: no), or when it is determined that analiasing-sensitive area is not continuous in the upward and downwarddirections from the target pixel (step S710: no), a determination ismade as to whether the processing described above has been completed forall the target pixels (step S716). When pixels remain that have not beenprocessed (step S716: no), the process returns to the beginning, asingle new target pixel is selected (step S702), and the sequence ofprocesses described above is performed for the selected target pixel.Since the determination is eventually made during the repetition of thisprocessing that processing has been completed for all the target pixels(step S716: yes), the block noise removal process shown in FIG. 30 iscompleted, and the process returns to the edge noise removal process ofModification 3 shown in FIG. 29.

When block noise is removed in the block noise removal process (stepS700) described above, the block noise is removed only in a sufficientlyinward area by determining whether the target pixel is locatedsufficiently inward from the vicinity of the boundary of thealiasing-sensitive area. It is therefore possible to reduce the dangerof creating new edge noise in the edge direction map when attempting toremove block noise.

After block noise is removed, it is particularly effective to performthe isolated noise removal process (step S300) again. The reason is thateven if there is edge noise that could not be removed by a singleexecution of the isolated noise removal process, removing the blocknoise increases the possibility that the edge noise will becomeremovable. Performing the isolated noise removal process (step S300)again therefore makes it possible to remove almost all of the remainingedge noise.

It is for this reason that the isolated noise removal process (stepS300) is performed again after the block noise removal process (stepS700) in the edge noise removal process (step S170) of Modification 3shown in FIG. 29. The isolated noise removal process is also notnecessarily performed only once, and may be performed a plurality oftimes, or the cluster noise removal process may also be performed afterthe isolated noise removal process.

After the edge noise removal process (step S170) of Modification 3 shownin FIG. 29 is completed as described above, the process returns to thecolor image data generation process shown in FIG. 3, and then processingis performed to interpolate missing color components in the raw imagedata (steps S108 through S112 in FIG. 3). Since the edge direction mapreferenced at this time is an accurate edge direction map from whichedge noise has been removed, the color components missing from the rawimage data can be appropriately interpolated to generate suitable colorimage data.

The digital camera 100 of the embodiment generates the color image datafrom the raw image data according to the procedure described above. Thepresent invention is not restricted to any of this embodiment and itsmodifications described above but may be attained in variousapplications and aspects without departing from the scope or spirit ofthe main characteristics of the present invention.

1. An image processing apparatus for receiving mosaic image data having settings of only one color component, R, G, or B, in each pixel and subjects the received mosaic image data to a series of image processing to generate color image data with settings of all the three color components R, G, and B in each pixel; wherein the mosaic image data is expressed as a Bayer color filter; and the image processing apparatus comprises: a vertical-direction color difference component computation module configured to obtain the G component in the pixel location of another color component or to obtain another color component in the G component pixel location using vertically neighboring available components in order to compute a vertical-direction color difference component between the G component and the another color component in each pixel included in each of the pixel columns of the mosaic image; a horizontal-direction color difference component computation module configured to obtain the G component in the pixel location of another color component or to obtain another color component in the G component pixel location using horizontally neighboring color components in order to compute a horizontal-direction color difference component between the G component and the another color component in each pixel included in each of the pixel rows of the mosaic image; an edge direction determination module configured to determine an edge direction in a pixel by selecting from among said mosaic image data a pixel in which said R component or said B component is set, and comparing an amount of variation of said vertical color difference component and an amount of variation of said horizontal color change component of the selected pixel; an edge direction map generation module configured to generate an edge direction map in which the edge direction is set in each pixel of said R component and each pixel of said B component, by storing said determined edge direction for each said selected pixel; an edge noise removal module configured to detect edge noise that is an erroneously determined edge direction by comparing said edge direction set in said edge direction map with the edge directions of surrounding pixels, and removing the edge noise; and a missing color component interpolation module configured to interpolate a missing color component, which is a color component not set in the pixel for each pixel of said mosaic image data, on the basis of said color component set in the mosaic image data while referencing said edge direction map from which said edge noise has been removed.
 2. The image processing apparatus according to claim 1, wherein said edge noise removal module comprises: target pixel setting module configured to select from among said edge direction map a single pixel in which said edge direction is set, and setting the pixel as a target pixel for determining whether the edge direction is said edge noise; an edge noise detection module configured to select two pixels in a position of a second pixel in said vertical direction from said target pixel, or two pixels in a position of a second pixel in said horizontal direction from the target pixel, and detecting the edge direction of the target pixel as said edge noise when said two edge directions set in the two pixels selected from any one of the vertical direction and the horizontal direction are the same as each other and different than the edge direction set in the target pixel; and an edge direction changing module configured to change said edge direction detected as said edge noise to a different edge direction.
 3. The image processing apparatus according to claim 1, wherein said edge noise removal module comprises: a target pixel setting module configured to select from among said edge direction map a single pixel in which said edge direction is set, and setting the pixel as a target pixel for determining whether the edge direction is said edge noise; a precondition determination module configured to select four pixels adjacent in a direction diagonal to said target pixel, and determining whether the pixels satisfy a prescribed precondition when said edge directions set in the selected four pixels are all the same as each other and different than the edge direction set in the target pixel; an edge noise detection module configured to select all pixels within the prescribed number of pixels in said horizontal direction from said target pixel that satisfies said precondition, or selecting all pixels within a prescribed number of pixels in said vertical direction, and detecting the edge direction of the target pixel as said edge noise only when all of said edge directions set in the selected pixels are not the same as said edge direction of the target pixel; and an edge direction changing module configured to change said edge direction detected as said edge noise to a different edge direction.
 4. The image processing apparatus according to claim 1, comprising: a target area extraction module configured to extract as a target area for removing said edge noise an area in which an amount of variation between color components having the same said mosaic image data is smaller than a prescribed first threshold value and larger than a prescribed second threshold value that is smaller than the first threshold value; wherein said edge noise removal module is a module configured to remove said edge noise with respect to said edge direction set in a pixel within said target area.
 5. The image processing apparatus according to claim 1, comprising: an edge noise re-removal module configured to re-remove the edge noise with respect to said edge direction map from which said edge noise has been removed.
 6. An image processing method comprising receiving mosaic image data having settings of only one color component, R, G, or B, in each pixel and subjecting the received mosaic image data to a series of image processing to generate color image data with settings of all the three color components R, G, and B in each pixel; wherein the image processing method comprises: a first step of receiving, as the mosaic image data, image data expressed as a Bayer color filter; a second step of obtaining the G component and the another color component in each pixel included in each of the pixel columns of the mosaic image data, and computing a vertical-direction color difference component between the G component and the another color component in each pixel included in each of the pixel columns in the vertical direction; a third step of obtaining the G component and the another color component in each pixel included in each of the pixel rows of the mosaic image data, and computing a horizontal-direction color difference component between the G component and the another color component in each pixel included in each of the pixel rows in the horizontal direction; a fourth step of determining an edge direction in a pixel by selecting from among said mosaic image data a pixel in which said R component or said B component is set, and comparing an amount of variation of said vertical color difference component and an amount of variation of said horizontal color change component of the selected pixel; a fifth step of generating an edge direction map in which the edge direction is set in each pixel of said R component and each pixel of said B component, by storing said determined edge direction for each said selected pixel; a sixth step of detecting edge noise that is an erroneously determined edge direction by comparing said edge direction set in said edge direction map with the edge directions of surrounding pixels, and removing the edge noise; and a seventh step of interpolating a missing color component, which is a color component not set in the pixel for each pixel of said mosaic image data, on the basis of said color component set in the mosaic image data while referencing said edge direction map from which said edge noise has been removed.
 7. A computer-readable medium having a program for causing a computer to actualize a method for receiving mosaic image data having settings of only one color component, R, G, or B, in each pixel and subjecting the received mosaic image data to a series of image processing to generate color image data with settings of all the three color components R, G, and B in each pixel, the program causing the computer to actualize: a first function of receiving, as the mosaic image data, image data expressed as a Bayer color filter; a second function of obtaining the G component and the another color component in each pixel included in each of the pixel columns of the mosaic image data, and computing a vertical-direction color difference component between the G component and the another color component in each pixel included in each of the pixel columns in the vertical direction; a third function of obtaining the G component and the another color component in each pixel included in each of the pixel rows of the mosaic image data, and computing a horizontal-direction color difference component between the G component and the another color component in each pixel included in each of the pixel rows in the horizontal direction; a fourth function for determining an edge direction in a pixel by selecting from among said mosaic image data a pixel in which said R component or said B component is set, and comparing an amount of variation of said vertical color difference component and an amount of variation of said horizontal color change component of the selected pixel; a fifth function for generating an edge direction map in which the edge direction is set in each pixel of said R component and each pixel of said B component, by storing said determined edge direction for each said selected pixel; a sixth function for detecting edge noise that is an erroneously determined edge direction by comparing said edge direction set in said edge direction map with the edge directions of surrounding pixels, and removing the edge noise; and a seventh function for interpolating a missing color component, which is a color component not set in the pixel for each pixel of said mosaic image data, on the basis of said color component set in the mosaic image data while referencing said edge direction map from which said edge noise has been removed. 